MolCA: Molecular Graph-Language Modeling with Cross-Modal Projector and Uni-Modal Adapter
Authors: Zhiyuan Liu
Sihang Li
Yanchen Luo
Hao Fei
Yixin Cao
Kenji Kawaguchi
Xiang Wang
Tat-Seng Chua
Venue: EMNLP 2023 Main (2023)
Keywords: Molecular Language Modeling
Cross-Modal Alignment
Molecule Captioning
Molecule-Text Retrieval

Abstract: Language Models (LMs) have demonstrated impressive molecule understanding ability on various 1D text-related tasks. However, they inherently lack 2D graph perception — a critical ability of human professionals in comprehending molecules' topological structures. To bridge this gap, we propose MolCA: Molecular Graph-Language Modeling with Cross-Modal Projector and Uni-Modal Adapter. MolCA enables an LM (i.e., Galactica) to understand both text- and graph-based molecular contents via the cross-modal projector. Specifically, the cross-modal projector is implemented as a Q-Former to connect a graph encoder's representation space and an LM's text space. Further, MolCA employs a uni-modal adapter (i.e., LoRA) for the LM's efficient adaptation to downstream tasks. Unlike previous studies that couple an LM with a graph encoder via cross-modal contrastive learning, MolCA retains the LM's ability of open-ended text generation and augments it with 2D graph information. To showcase its effectiveness, we extensively benchmark MolCA on tasks of molecule captioning, IUPAC name prediction, and molecule-text retrieval, on which MolCA significantly outperforms the baselines.

MolCA: Molecular Graph-Language Modeling with Cross-Modal Projector and Uni-Modal Adapter
Zhiyuan Liui Sihang Li*
Yanchen Luo
Hao Fei
Yixin Cao Kenji Kawaguchi" Xiang Wang
Tat-Seng Chua!
National University of Singapore, UUniversity of Science and Technology of China SSingapore Management University
(acharkq, sihang0520, haofei37ènus, edu. {kenji, chuats Sociuatssccone fécomp. nus. edu. sg
(rthann@222,1uoy088888838 luoyc0830, caoyixin2011 , xiangwang (etp:/aaann233]@mmmmmm 223)amm
Abstract
Language Models (L.Ms) have demonstrated impressive molecule understanding ability on various ID text-related tasks. However, they inherently lack 2D graph perception a crit- ical ability of human professionals compre- hending molecules' topological structures. To bridge this gap, we propose MolCA: Molecular Langua-e Modeling with Cross-Modal Projector and Uni-Modal Adapter. MolCA ables an LM (i.e., Galactica) to understand both text- and graph-based molecular contents via the cross-modal projector. Specifically, the cross-modal projector is implemented as a Q- Former connect graph encoder's repre- sentation space and an LM's text space. Fur- ther, MolCA employs uni-modal adapter (iee. LoRA) for the LM's efficient adaptation to downstream tasks. Unlike previous studies that couple an LM with a graph encoder via cross-modal contrastive learning, MolCA re- tains the LM's ability of open-ended text gen- eration and augments it with 2D graph infor- mation. To showcase its effectiveness, extensively benchmark MoICA on tasks of molecule captioning, IUPAC name prediction, and molecule-text retrieval, on which MolCA significantly outperforms the baselines. Our codes and checkpoints be found at https: //gi thub. com/ acharkq/MolCA.
nAI, 2023). Therefore, becomes increasingly gent to incorporate these LMS to augment research in chemistry and biology.
For this purpose, we aim to utilize LMs for molecule understanding. As shown in Figure la, most existing LMs (Touvron et al., 2023; Zhang et al., 2022; Zeng et al., 2022) represent molecules by their ID Simplified Molecular Input Line Entry System (SMILES) strings (Weininger, 1988) and process them in manner similar to texts. While convenient, treating molecules as strings overlooks the molecules' 2D graph representations, which are crucial to human professionals in comprehending the molecule structures (Wells, 2012). To com- bat that, recent works (Su et al., 2022; Liu et al., 2022b) represent molecules as graphs and use Graph Neural Network (GNN; Xu al., 2019) the molecular graph encoder. The graph encoder trained jointly with an LM through cross-modas contrastive learning (Radford et al., 2021; Li al., 2022), as illustrated Figure lb. However, the ap- plication scope of cross-modal contrastive learning is limited (Alayrac et al., 2022): it is suitable for retrieval tasks, but is insufficient for open-ended molecule-to-ter generation tasks, such as molecule captioning (Edwards et al., 2022) and molecule's IUPAC name prediction (Taylor al., 2022). This because molecule-to-tey. generation tional generation task (Keskar et al., 2019; Raffel et al., 2020). requires the LM to understand 2D graphs as the generation conditions, which con- trastive learning cannot achieve. Su et al. (2022) attempt to directly input 2D graphs' representations into LMs, however showing limited improvement.
1 Introduction
condi-
Language Models (LMs) have demonstrated signif- icant achievements across various domains (Devlin et al., 2019; Zhao et al., 2023). Notably, the wealth biochemical literature in LMs' pretraining data has enabled LMS to obtain a high-level understand- ing of biochemical concepts and molecule prop- erties. This can be reflected by their promising performances in biochemical and medical question- answering benchmarks (Taylor al., 2022; Ope-
To bridge this gap. devise MolCA: Molecular Graph-Language Modeling with Cross-Modal Pro- jector and Uni-Modal Adapter. MolCA enables the LM to understand 2D graphs as inputs, therefore effectively conditioning the molecule-to-tex gener-
"ooresponndigg author. Xiang Wang is also affiliated with Institute of Artificial Intelligence, Institute of Dataspace, Hefci Comprehensive National Science Center.

---

Graph
Encoder
non Appaace sakonloo witu
201-00001-00000-C10-000
Input
Cross Modal Projector
Motal Acapor
conojactedo
Language
Language
Input
Encodar
Language Model
Generate
Aoetytosteglc nod spowars
Contrant
Generate
Asstyfudcyple odtrisss white eryitblee
(o) language modelligg
(b) Cross-modal contrastive learning.
(c) MolCA.
Figure 1: Comparison of molecular language modeling methods.
Stage
Model Architecture
Downstreom Task
fine-tuned for downstream generation tasks.
Pretrain stage
Graph Eascoor
Frnjenoor
modal retrieval
Our contributions can be summarized as follows:
Protrain stage
Gragh
Languoge
We propose MolCA, a pioneering method for molecular language modeling. MolCA enables an LM to perceive 2D molecular graphs, thereby facilitating molecule-to-text generation tasks.
Generation tasks:
Fine-tune stage
Gragh Erscowy
Cross Modal Proecca
Languago
Malccuee caplion,
IUPAC name prediction
Figure 2: MolCA's three-stage training pipeline.
ation process. To enable the LM to understand 2D graphs, we identify that the key challenge is cross- modal alignment (Li et al., 2023; Merullo et al., 2023; Alayrac al., 2022): translating the repre- sentations of 2D graphs into ID soft prompts (Li and Liang, 2021) in the text space so that the LM can understand. This translation is facilitated by the cross-modal projector, bridging the gap between the graph encoder's representation space and the LM's input space, as illustrated in Figure 1. Specif- ically, we implement the cross-modal projector as a Q-Former (Li et al., 2023) due to its effective- ness in vision-language tasks. With an effective cross-modal projector,
MolCA sets new state-of-the-arts in a variety of benchmarks. It surpasses the baselines by 4.0 and 8.7 BLEU-2 for molecule captioning on CheBI-20 (Edwards et al., 2022) and our curated PubChem324k dataset, respectively. Moreover, in predicting IUPAC names, MolCA shows a significant advantage of 10.0 BLEU-2 over the baselines. For molecule-text retrieval, MolCA outperforms the baselines by 20% retrieval ac- curacy in PubChem324k and achieves the best performances in PCDes (Zeng et al., 2022) and MoMu datasets (Su et al., 2022).
We conduct ablation studies to show MolCA's effectiveness of incorporating 2D graphs into LMs for molccule-related tasks. Additionally, quantitative analysis shows that incorporat- ing 2D graphs helps improve the LM's ability count functional groups inside molecules.
harness the power of existing large LMs (Taylor et al., 2022; Tou- vron et al., 2023) for molecule-to-ter generation. However, given a large LM with billion scale pa- rameters, its efficiency of downstream fine-tuning arises as a new problem. Therefore, we integrate the LM with uni-modal adapter, i.e., LoRA (Hu et al., 2022), to enable its efficient adaptation.
Model Architecture
Here introduce three key components of MolCA's architecture: 1) a graph encoder for 2D structure understanding, 2) an LM for text gener- ation, and 3) a cross-modal projector to connect the graph encoder and the L.M. We describe the uni-modal adapter in Section 3.3.
As Figure illustrates. MolCA uses three-stage training pipeline integrate its components. The two pretrain stages aim to develop the cross-modal alignment ability of the cross-modal projector. In pretrain stage 1, the projector and the encoder are trained to extract the molecule features that are the most relevant to the text. This stage endows the resulting model with powerful molecule-text retrieval ability. In pretrain stage 2, the cross-modal projector is connected to frozen LM and trained for molecule captioning. This task forces the cross- modal projector to produce soft prompts that the LM can understand, In the final stage, MolCA is
Graph Encoder. Given the rich structural pat- terns in molecules, we leverage GNN-based en- coder encode molecular graphs. Specifically, employ five-layer GINE (Hu al., 2020) that pretrained 2 million molecules from the ZINC15 (Sterling and Irwin, 2015) dataset by con- trastive learning (You et al., 2020). Given a molec- ular graph g. the graph encoder f can generate

---

Malccuta contrastng
Molezulo-soxi
Malccule capsoning
matching
Fund-forwant
Fleed-forwart
food-forward
Fued-ffrrwad
Fund-farward
Food--orwarl Forward
Graph Emoccerr
Cress ettendon
Cross ottention
sttendio
Sel
attention
Soft omontion
Causal solf
Nx
Input 20 molecule
Cross-Modal Projector: Q-Former é
Query takema
(CLB) Acttrootioll
Input 1D text
PosT appears
Figure 3: MolCA's pretrain stage The graph encoder and the cross-modal projector (i.e., Q-Former) jointly optimized using three cross-modal tasks. Modules of the same color share weights.
structure-aware features for every node of
tokens can interact with the text inputs through the same self-attention modules. Note that, the query (1) tokens and text inputs are processed by different feed-forward networks, in order to maintain capac- ities for processing molecules and texts.
( Z E R xd
where lg| denotes the number of nodes in g. Language Model. To achieve effective text gen- eration performance, we employ Galactica (Tay- lor et al., 2022) as the base LMM. Galactica pre- trained on a large collection of scientific litera- ture, which encompasses fields like chemistry, bi- ology, and medicine. Its promising performance in text-based science question-answerinn marks (Hendrycks al., 2021; Jin al., 2019) derscores its understanding of high-level biochem- ical concepts. Notably, Galactica can process ID SMILES of molccules, which can potentially ben- efit our downstream tasks. Galactica is a decoder- only transformer LM based on the OPT (Zhang et al., 2022) architecture.
We initialize Q-Former from Sci-BERT (Belt- agy al., 2019), encoder-only transformer pretrained scientific publications. Q-Former's cross-attention modules are randomly initialized.
3 Training Pipeline
bench-
This section delves into the details of MolCA's three-stage training pipeline (ef. Figure 2). The two pretrain stages leverage dataset of molecule- text pairs D ((g) #1), (92, y2). train the cross-modal projector and the graph encoder. The goal of pretraining is to translate 2D molecular graphs into soft prompts that a frozen LM can un- derstand. The fine-tune stage focuses on efficient adaptation to downstream generation tasks.
Cross-Modal Projector. We implement the cross-modal projector Querying-Transforng (Q-Former) (Li et al., 2023) to map the graph en- coder's outputs to the LM's input text space. As shown Figure: 3, Q-former has different pro- cedures for processing 2D molecular graphs and ID texts. Given text inputs, Q-Former inserts [CLS] tokens at the beginning and processes the texts by N layers of self-attention modules and feed-forward networks. The self-attention modules adopt causal masks (Raffel al., 2020) when the pretraining task is text generation. On the other hand, given a molecular graph g. Q-Former works as a molecule feature extractor. Specifically, maintains set of learnale query tokens { ) inputs. These query tokens interact with the graph encoder's output through the cross- attention modules (Vaswani al., 2017) and extract molecule features. The cross-attention modules are added every two layers. Additionally, the query
3.1 Pretrain Stage 1: Learning to Extract Text Relevant Molecule Representations
In this stage, we aim to optimize the cross-modal projector (i.e., Q-Former) to extract the molecule features most relevant to the text input. This stage serves as a "warmup" training for the cross-modal projector before connecting to the LM. Inspired by BLIP2 (Li et al., 2023), we simultaneously ap- ply three cross-moue modsl pretraining tasks that are tailored for Q-Former's architecture: molecule-text contrasting, molecule-text matching, and molecule captioning. These pretraining tasks endow the Q- Former with strong molecule-text retrieval ability. Therefore, save the resulting model from this stage for downstream retrieval tasks. We now elab- orate on the three pretraining tasks.
Molecule- Text Contrasting (MTC). We apply

---

cross-modal contrastive learning (Radford al., 2021) to train the Q-Former extract text-revelant molecule features. In this task, query tokens and text inputs are fed into the Q-Former separately (left of Figure 3) to obtain Q-Former's molecule representations and text representations.
Therefore, in retrieval experiments, we use MTC to first retrieve the top k samples and use MTM for re-ranking, thereby improving the performance.
Molecule Captioning (MCap). MCap aims to generate the molecule's ' text description based on the molecule representations. For this task, we adopt special masking strategy in self-attention modules to ensure that the queries learn to ex- tract molecule features that correspond to the text descriptions. Specifically, we employ the bi- directional self-attention masks for queries, allow- ing them to see each other but not the text tokens. Further, we apply causal masks for texts on the same self-attention module to perform autoregres- sive decoding of text descriptions. Each text token can see the queries and the preceding text, but not the subsequent text tokens. Since the text tokens cannot directly interact with the graph encoder, they must obtain molecule information from the queries, forcing the queries to extract molecule information through the cross-attention modules. Let p1 (1 (g) be the probability of Q-Former generating text y for a graph g. We use the following loss function:
Formally, let (91. 91) ...,, (9B. }/) be a batch of molecule-text pairs. We denote g;'s 's Q-Former representations as { am k=1 (each element for one query token), and denote y, 's Q-Former representa- tion as t; (representation of the [CLS] token). For arbitrary i,J E [1, BJ. we measure the similarity between t; and {m, No by computing the max- imum similarity between tę and every element in Ng
(m The MTC loss Exre can be written as:
exp(maxx cos t:)/T) Σ log
exp(maxx cos(mms.. (mik;; tz)/T)
B
exp(maxx cos(t,, /T)
lig
log
exp(maxx (t,, m mjk)
1=1 1=1
MMTC = B L22
- fee
(2)
B
B
where c/ is the temperature-scaled cosine similarity. Temperature r is empirically set to 0.1. Molecule-Texx Matching (MTM). MTM is a bi- nary classification task, aiming to predict whether a molecule-text pair is matched (positive) or un- matched (negative). As Figure 3 illustrates, MTM allows the queries and the texts to interact through the same self-attention module. In this way, the queries can extract mudti-modal information from both molecules and texts. For MTM prediction, we attach a linear classifier after the mean pooling of all queries' Q-Former representations. Let p(g. v) denotes MTM's predicted probability that (g. 3) is matched. MTM loss (MM can be written as:
(MCap
Σ
log (v.li))
(4)
B
3.2 Pretrain Stage 2: Aligning 2D Molecular
Graphs to Texts via Language Modeling
In this stage, aim to align the cross-modal pro- jector's outputs to the text space of frozen LM. As Figure 4 illustrates, we feed the cross-modal projector's s representations of 2D molccular graphs to the frozen LM as inputs, and train the model to generate molecules' text descriptions. This process encourages the cross-modal projector to provide representations that the LM can understand, so to prompt the text generation. Additionally, we also use a molecule's ID SMILES guide the generation (cf. Figure 4). This is because most LMs (Taylor et al., 2022: Touvron et al., 2023; Zhang et al., 2022) use SMILES during pretraining. Therefore, these LMs have established some cor- relations between SMILES and their text contexts. Thus, including SMILES can potentially prompt the corresponding biochemical knowledge. On the other hand, incorporating 2D graphs can help cap- ture structural patterns that are hard learn from ID SMILES. We will show later in experiments that combining 2D graphs and ID SMILES can boost performance.
B
1
4MTM = B (E
Σ
log p(gs, ve)+
1=1
log plgs. ,,) + log p(gn: H.)].
(3)
where U(1, B) is a uniform distribution; y and gle are random negative samples in batch.
Similar to MTC, MTM also computes the simi- larity between molecule-text pairs. The difference is that MTM can capture more fine-grained simi- larity between a molecule and a text through the self-attention and cross-attention modules, com- pared to the simple cosine similarity used by MTC.

---

Graph Encoder
Grass Modal Projector
Greph Encoder
Cross-Modal Projector
CCI-DIDC1=CD =CD=GTG/=OOO
Input
eoy-ojoGracc
+D0+D10/=010
Thu moieasteia JUPAC narms
Input
Langiaage
Language Model
Uni-Modal Adapter: LoRA
Generate
Aooptsaltyyic ond appoars adorioso wrill oystals
Generate
Powrallyrayybeeenic.cc and
Figure 4: MolCA's pretrain stage 2 by molecule captioning.
Figure MolCA's fine-tum stage for molecule to-text generation. The example shows the prediction of molecule's IUPAC name.
Formally, consider a molecule-text pair (g. 3) and g's SMILES repsentation 8, The cross-modal projector representations of g are denoted as {m&) We define p2(-) as the text distribution parameterized by the frozen LM. We optimize the cross-modal projector and the graph encoder by minimizing the following loss function:
Subset
Size
Avg mol len
Min text len
Avg text len
Pretrain
309689
35
1
18
Thain
12000
32
20
60
Valid
1000
32
20
61
Test
2000
31
20
60
Table Statistics of the PubChem324k dataset. We count the text length by splitting the text spaces.
log p2 (2)223) (m
min(d1 d2). LoRA can effectively adapt the LM to downstream tasks while requiring little memory overhead for storing gradients.
log po( 1-11 (m 8). (5)
4
Experiments
3.3 Fine-tune Stage: Uni-Modal Adapter for Efficient Downstream Adaptation
4.1
Experimental Setting
Here we briefly present the experimental settings. More details can be found in Appendix B. PubChem324k Dataset. We collect PubChem- 324k dataset containing 324k molecule-text pairs from the PubChem website!. Table presents the dataset statistics. Notice that, the dataset in- cludes many uninformative texts, such as "The molecule is a peptide". Therefore, we sample a high-quality subset of 15k pairs with text longer than 19 words for downstream tasks. This high- quality subset is further randomly divided into the train/valid/test sets. The remaining dataset, which is more noisy, is used for pretraining.
In this stage, we fine-tune MolCA for downstream generation tasks. As Figure illustrates, we ap- pend a text prompt of the task description after the molecule representations. Then, we apply language modeling loss to fine-tune MolCA for generation tasks, such molecule's 's IUPAC name prediction. Uni-Modal Adapter. In MolCA, the LM is ac- counted for large portion of computation over- head: it can have 1B parameters, while the cross- modal projector and graph encoder only have total of -..01B1 parameters. Therefore, we employ a uni-modal adapter for the LM's efficient adaptation to downstream tasks. Specifically, we employ the LoRA (Hu et al., 2022) adapter due to its simple implementation and promising performances (Liu et al., 2022a). As shown in Figure 5, for selected weight matrices (e.g. W E xd2) in the LM, LoRA adds pairs of rank decomposition matrices (e.g., BA, B RRdixr A с (Rrxda) in parallel to them. The original h = Wa layer is changed to:
Baselines. For generation tasks, we compare MolCA with the following baselines: T5 (Raffel et al., 2020), MolT5 (Edwards et al., 2022), and MoMu (Su al., 2022). For molecule-text retrieval. we also include these methods: MoleculeSTM (Liu et al., 2022b), KV-PLM (Zeng et al., 2022), and Sci-BERT (Beltagy et al., 2019).
4.2 Molecule Captioning
h Wx BAæ,
(6)
We evaluate MolCA for molecule captioning the datasets of PubChem324k and CheBI-20 (Edwards
where W kept frozen and the newly added BA is trained during adaptation. Given a small r
hhttps:/pubbhhen.nn ncbi, nln, niv.

---

Model
#Trainable params
BLEU-2
BLEU-4 ROUGE-1
ROUGE-2 ROUGE-L METEOR
ID SMILES
MolTS-Small
80M, full
14.8
8.5
26.5
13.5
23.6
18.5
MolTS-Buse
250M, full ft
30.1
20.9
40.3
25.1
33.8
35.6
MolgS-
780M, full ft
30.2
22.2
41.5
25.9
34.8
36.6
1D SMILES 2D Graph
MoMu-Small
82M, full
19.1
12.0
29.7
16.3
26.7
21.8
MoMu-Base
252M. full ft
30.2
21.5
40.5
25.1
34.4
34.2
MoMu-Laarge
782M, full ft
31.1
22.8
41.8
25.7
36.7
36.2
MolCA, Morge-
877M. full ft
33.7
27.0
49.7
35.6
44.4
42.4
MolCA, GalaC125M
222M, full ft
32.4
24.9
44.9
30.1
39.5
39,2
MolCA, Galaci.s
100M, LoRA ft*
39.8
31.7
51.7
37.3
46.2
46.8
(a)
PubChem324k dataset. Baseline performances are reproduced using their source codes (Edwards et al., 2022; Su et al., 2022).
Model
#Trainable params
BLEU-2 BLEU-4
ROUGE-1
ROUGE-2
ROUGE-L METEOR
ID SMILES
T5-Small
80M, full
50.1
41.5
60.2
44.6
54.5
53.2
T5-Base
250M, full ft
51.1
42.3
60.7
45.1
55.0
53.9
T5-Large
780M, full
55.8
46.7
63.0
47.8
56.9
58.6
MolT5-Small
80M, full
51.9
43.6
62.0
46.9
56.3
55.1
MolTS-Base
250M, full
54.0
45.7
63.4
57.8
56,9
MolTS-Large
780M,
59.4
50.8
$1.0
59.4
61.4
ID SMILES 2D Graph
MoMu-Small
82M. full
53.2
44.5
56.4
55.7
MoMu-Base
252M. full ft
54.9
46.2
57.5
57.6
MoMu-Large
782M. full ft
59.9
51.5
59.3
59.7
MolCA, Galac125M
222M. full ft
61.6
52.9
67.4
53.3
61.5
63.9
MolCA, Galacı.5a
110M, LoRA ft*
63.9
55.5
69.7
55.8
63.6
66.9
(b) CheBI-20 datuset. Baseline performances are borrowed from their original papers (Edwards et al., 2022; Su et al., 2022).
Table Performances (%) of molecule captioning on PubChem324k and CheBI-20 datasets. Bold indicates the best performance and underline indicates the second best performance, Full ft denotes full parameter fine-tuning, "The LoRA configurations for PubChem324k and CheBI-20 datasets are different. Details are in Appendix B.
et al., 2022). Specifically, we implement MolCA with the base LMs of Galacticaj,sg Galactica|25M: and MolT5-Large. We employ full parameter fine- tuming for GalacticarzsM and MolT5-Large due to their smaller scales. We fine-tune MolCA and basc- lines the dataset's training and report the test set performance selected by the valid set. Follow- ing (Edwards et al., 2022), we adopt BLEU (Pap- ineni et al., 2002), ROUGE (Lin, 2004), and ME- TEOR (Banerjee and Lavie, 2005) as the evaluation metrics. As shown in Table 2, we observe that:
ing system for chemical compounds, known as IU- PAC names (Favre and Powell, 2013). Notably, this naming system relies on identifying specific molecule structures, including hydrocarbon chains and double/triple bonds. Therefore, correctly pre- dicting IUPAC names indicates a model's profi- ciency understand molecule structures. We fine-tune MolCA and baselines using the Pub- Chem324k's ' training set to generate a molecule's IUPAC name. As shown in Table 3, MolCA consis- tently outperforms the baselines by large margin of 10.0 BLEU-2, highlighting MolCA's advantage in comprehending molecule structures.
1. MolCA consistently outperforms the base- lines by large margin. Specifcally, MolCA, Galac1.3 achieves the highest performance on all metrics. outperforms the baselines by 8.7 BLEU- 2 on PubChem324k and 4.0 BLEU-2 on CheBI-20. 2. MolCA, GalacI25M outperforms baselines of larger sizes across all metrics, showing that MolCA's advantage is not limited to model scale.
4.4 Molecule-Tee Retrieval
We evaluate MolCA for molecule-text retrieval on the datasets of PubChem324k, PCDes (Zeng et al., 2022) and MoMu (Su al., 2022). Specifically, we evaluate MolCA's checkpoint from pretrain stage 1 without further fine-tuning. For all experiments. MolCA first retrieves the top 128 candidates us- ing MTC, then employs the MTM module for re- ranking. We select Accuracy (Acc) and Recall 20
4.3 IUPAC Name Prediction
The International Union of Pure and Applicd Chem- istry (IUPAC) has established a standardized nam-

---

Model
#Trainable params
BLEU-2
BLEU-4
ROUGE-1
ROUGE-2
ROUGE-L
METEOR
1D SMILES
MolT5-Small
80M, full
48.6
35.2
40.0
16.1
34.3
42.5
MolTS-Base
250M. full ft
52.7
41.5
50.7
26.0
44.3
53.2
Large-Large
780M, full
59.4
49.7
55.9
33.3
58.5
1D SMILES 2D Graph
MolCA, GalacI25M
222M, full ft
72.9
65.1
69.5
48.0
62.6
71.6
MolCA, Galacy,3 100M, LoRA ft
74.6
66.1
70.5
49.1
64.2
73.0
Table 3: Performances (%) of predicting molecule's IUPAC names the PubChem324k dataset.
Baseline
performances are obtained by running their source codes (Edwards et al., 2022).
M2T
T2M
MolCA's performance. This can be attributed to MTM's ability to model rong-range interactions between molecule features and texts, achieved by the cross-attention and self-attention modules.
Model
Acc
R@20
Acc
R @ 20
ID SMILES
Sci- BERT
39.3
86.1
37.9
85.1
KV-PLM
38.8
86.3
38.7
85.6
3. MolCA's good performances tially attributed our larger pretrain dataset Pub- Chem324k. As shown in Table 4a, we compare the performances of MoMu's original checkpoint (pretrained on 15k molecule-text pairs) with our reproduced MoMu using PubChem324k, The latter improves the retrieval accuracy by over 25%.
be par-
2D Graph
MoMu-S#
41.2
43.6
MoMu-K*
11.3
41.0
39.9
MoMu-S
86.5
86.5
MoMu-K
41.8
88.9
42.4
88.5
MoleculeSTM
47.1
89.0
45.4
91.5
MolCA w/o MTM
93.7
92.3
MolCA
69.4
95.7
69.6
94.6
Performances (%) in the PubChem324k dataset.
PCDes dataset
MoMu dataset
4.5
Ablation Study on Representation Types
Model
M2T
T2M
M2T
T2M
Here ablate the two representations types of molecules: ID SMILES and 2D graphs. We com- pare MolCA with its two variants: 1) ID SMILES: an LM that uses only ID SMILES for pretraining and fine-tuning. For a fair comparison, we pretrain this variant on PubChem324k 's pretrain subset for molecule captioning before its downstream adapta- tion; 2) 2D Graph: this variant follows the original MolCA's training pipeline, except not using ID SMILES in pretrain stage 2 and fine-tune stage.
1D SMILES
Sci- !
60.7
60.8
0.3
0,3
KV-PLM
75.9
64.3
0.5
0.3
2D Graph
MoMu-St
79.1
75.5
43.3
43.4
MoMu-Kt
80.2
79.0
43.7
43.5
MoleculeSTM
84.6
85.1
75.8
74.5
MolCA w/o MTM
88.0
85.5
81.5
81.6
MolCA
90.5
87.6
88.6
87.3
(b) Recall @0 (%) in the PCDes and MoMu datasets.
Table 4: Molecule-text retrieval performances. We re- port performances of using molecule to retrieve text (M2T) and using text to retrieve molecule (T2M). de- notes performance evaluated on the baseline's 's released checkpoint. & denotes result borrowed from (Su et al.. 2022). Other models are trained on PubChem322k's pretrain subset. The complete results in Appendix
End Task Ablation. Table 5 presents the sults for molecule-to-ter generation and molecule property prediction (Hu al., 2020) tasks. We observe that combing 2D graphs and ID SMILES leads to improved performance in all the compared tasks. This demonstrates MolCA's effectiveness incorporating molecules' 2D graph representations. Counting Functional Groups (FGs). We ablate MolCA's capability of counting 85 types of FGs inside molecules. An FG is a molecule's 's subgraph that exhibits consistent chemical behaviors across different molecules (Rong et al., 2020). Correctly counting FGs can help understand molecule's properties. As shown in Figure 6, incorporating 2D graphs significantly improves MolCA's perfor- mance in counting FGs, thereby enhancing its abil- ity in understanding molecule structures.
(R (2) as the evaluation metrics, and report the performances of retrieval in the entire test set. As shown in Table 4, we observe that:
1. MolCA demonstrates superior performance over baselines. Specifically, in PubChem324k, MolCA improves the accuracy by more than 20% over the baselines. In PCDes and MoMu, MolCA also consistently outperforms the baselines. demon- strating its effectiveness for molecule-text retrieval. Incorporating MTM significantly improves

---

Representation type
BLEU-2
BLEU4
ROUGE-1
ROUGE-2
ROUGE-L
METEOR
Molecule Captioning, PubChem324k
ID SMILES
33.7
26.0
45.4
31.6
40.7
40.3
2D Graph
35.7
27.4
47.3
32.3
41.8
42.0
1D SMILES 2D Graph
39.8
31.7
51.7
37.3
46.2
46.8
Molecule Captioning, CheBI-20
ID SMILES
58.3
49.4
65.9
51.3
59.7
62.4
ID SMILES + 2D Graph
63.9
55.5
69.7
55.8
63.6
66.9
IUPAC Name Prediction, PubChem324k ID SMILES ID SMILES + 2D Graph
70.7
60.7
68.6
46.2
61.7
71.5
74.6
66.1
70.5
49.1
64.2
73.0
(a) Ablating the representation type on tasks of molecule captioning and IUPAC name prediction.
Representation type
Bace
BBBP
ClinTox
ToxCast
Sider
Tox21
Mean
ID SMILES
79.3+0.8
70,8+0.6
89.0=1.7
56.2+0.7
61.1=1.2 63.0#1.7
76.0+0.5
72.1
ID SMILES 2D Graph
79.8+0.5
70.0+0.5
89.5+0.7
64.5.0.8
77.2+0.5
74.0
(b) ROC-AUC (%) scores on six molecule property prediction datasets from MoleculeNet (Wu et al., 2018). We use scaffold split following (Hu et al., 2020). We report the performance's 's mean values and standard deviations across three random seeds.
Table 5: Ablating molecule's representation types. All compared models fine-tune the base LM of Galactica
1.0
loss on ID SMILES. Vaucher et al. (2021) pro- pose to predict the chemistry experiment actions by reading chemical reaction equations. MolT5 (Ed- wards et al., 2022) presents several T5-based (Raf- fel et al., 2020) LMs for SMILES-to-text and text- to-SMILES translations. Further, Christofidellis et al. (2023) propose fine-tune T5 for chem- ical reaction prediction and retrosynthesis tasks. MolCA different from these methods that exclu- sively utilize 1D SMILES to represent molecules. Instead, MolCA aims to enable LMs to perceive molecules' 2D graph representations.
SMILES
SMILES 20 Graph
RASE
0.4
Mas
0.5
0.3
0.0
50
epochs
spochs
(a) Training loss.
(b) Valid performance.
Figure Ablating MolCA for counting FGs inside molecules from PubChem324k. We plot average values across three random seeds. Blue shades indicate the range - one standard deviation. Evaluation metric is Root Mean Square Error (RMSE). Lower value indi- cates better performance.
5 Related Works
Molecule-Text Contrastive Learning. Driven by the demand a molecule-text retrieval system, Text2Mol (Edwards et al., 2021) employs cross- modal contrastive learning to train a molccular graph encoder of GCNs (Kipf and Welling, 2017) and text encoder of Sci-BERT (Beltagy et al.. 2019). Subsequent works (Su al., 2022; Liu al., 2022b; Seidl al., 2023) have proposed en- hancements, including the addition of inter-modal contrastive learning loss (Su et al., 2022) and apply- ing the model for text-based molccule editing (Liu et al., 2022b). However, cross-modal contrastive learning is unsuitable for open-ended conditional generation task (Alayrac et al., 2022), because of its focus learning similarity function. To resolve the problem, propose MolCA to enable the LM's understanding of 2D molecular graphs, facili- tating MolCA's capability of open-ended molecule- to-text generation.
Here we briefly review the molecule-related lit- crature. We discuss MolCA's relations to vision- language pretraining methods in Appendix A. Molecule Understanding via 1D Language Modeling. Due to the extensive biochemical liter- ature in their training corpus, some open-domain LMs (Zhang et al., 2022: Touvron et al., 2023; Chowdhery et al., 2022) have obtained a high-level understanding of molecular and chemical concepts. This is demonstrated through their promising per- formances in text-related biochemical and medical question-answerte benchmarks (Hendrycks et al., 2021; Jin al., 2019). Among these LMs, Galac- tica (Taylor al., 2022) shows competitive perfor- mances for using corpus that is primarily com- posed of scientific literature. Focusing on the chem- istry domain, KV-PLM (Zeng et al., 2022) models molecules by applying masked language modeling

---

6 Conclusion and Future Works In this work, propose MolCA, novel molec- ular language modeling method. MolCA aims to enable LMs to perceive 2D graphs for molecule-to- text generation. For this purpose, MolCA features a cross-modal projector to map representations of 2D graphs into the text space of LMs. It also em- ploys uni-modal adapter for efficient downstream adaptation. MolCA achieves statc-of-the-un perfor- mances molecule captioning and molecule-text retrieval benchmarks. Looking forward, we are interested in exploring LMs for 3D molecular mod- eling and drug discovery tasks.
Acknowledgement
This research is supported by NExT Research Cen- ter. This material is based upon work supported by the Google Cloud Research Credit program with the award search supported by the National Natural Sci- ence Foundation of China (9227010114) 14) and the University Synergy Innovation Program of Anhui Province (GXXT-2022-040).
CF7K-3AG44
This
References
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring. Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob L. Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals. Andrew Zisserman, and Karén Si- monyan. 2022. Flamingo: visual language model for few-shot learning. In NewrIPS.
Limitations
This work focuses on utilizing LMs' generation ability for molecule-text tasks. Other interesting abilities of LMs, like in-context learning and chain- of-thought reasoning, are beyond the scope of this research. We leave that to future exploration.
While MolCA offers improvements over base- lines, we observe that the current performance in molccule captioning is not yet sufficient for practical application. This can be attributed to the scale of pretraining data. To our knowledge, our PubChem324k dataset is the largest dataset of molecule-text pairs. However, compared to the 10M scale dataset (Changpinyo et al., 2021) for vision-language pretraining, dataset, consists of 324k data points, is comparatively smaller and limits the model's performance. Remedy solutions may include mining weakly supervised data from biochemical literature.
Satanjeev Banerjee and Alon Lavie. 2005. METEOR: an automatic metric MT evaluation with improved correlation with human judgments. In IEEvalua- rion @ ACL. pages 65-72. Association for Computa- tional Linguistics.
Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. Scibert: A pretrained language model for scientific text. In F.NNIP/IJCNLPP (7), pages 3613-3618. Association for Computational Linguistics.
Soravit Changpinyo, Piyush Sharma. Nan Ding, and Radu Soricut. 2021. Conceptual 12m: Pushing web- scale image-text pre-training to recognize long-tail tail visual concepts. In CVPR, pages 3558-3568. Com- puter Vision Foundation IEBE.
Broader Impacts
Our work has established new state-of-the-art per- formances in molecule captioning and molecule- text retrieval. It has broader impacts in two as- method
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint wrXiv:2204.02311.
pects: for chemistry professionals, of molecule captioning and molecule-text retrieval could be useful tools, potentially speeding up their research process; 2) for individuals without spe- cialized chemistry knowledge, our method could provide a more affordable way to access the basic chemical information of molecules.
Dimitrios Christofidellis, Giorgio Giannone, Jamnis Born, Ole Winther, Teodoro Laino, and Matteo Man- ica. 2023. Unifying molecular and textual represen- tations via multi-task language modelling. In ICMI..
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang. Boyang Li, Pascale Fung, and Steven Hoi. 2023. structblip: Towards general-purpor models with instruction tuning, In NeurIPS.
Our model shares the risks of most LMs. It can generate inaccurate information and can potentially be abused produce biased content. Further, con- sidering the limited scale of our training data, we strongly advise strictly testing our model before applying it real applications.
languae
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of

---

deep bidirectional transformers for language under- standing. In NNACL-HLI pages 4171-4186. Associ- ation for Computational Linguisties.
Thomas N Kipf and Max Welling. 2017. Semi- supervised classification with graph convolutional networks. In ICLR.
Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zong- han Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et 2022. Delta tuning: A comprehensive study of parameter efficient meth- ods for pre-trained language models. arXiv preprint arXiv:2203.06904
Greg Landrum. 2013. Rdkit documentation. Release. 1(1-79):4.
Junnan Li, Dongxu Li. Silvio Savarese, and Steven C. H. Hoi. 2023. BLIP-2: bootstrapping language-ime -mage pre-training with frozen image encoders and large language models. CoRR, abs/2301.12597.
Carl Edwards, Tuan Manh Lai, Kevin Ros, Garrett Honke, Kyunghyun Cho, and Heng Ji. 2022. Trans- lation between molecules and natural language. In EMNLP, pages 375 413. Association for Computa- tional Linguistics.
Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. ACL/IJCNLP (1), pages 4582-4597. Association for Computational Linguistics.
Carl Edwards, ChengXiang Zhai, and Heng Ji. 2021. Text2mol: Cross-modal molecule retrieval with natu- ral language queries. In EMNLP (1), pages 595-607. Association for Computational Linguistics.
Yangguang Li. Feng Liang, Lichen Zhao, Yufeng Cui. Wanli Ouyang, Jing Shao, Fengwei Yu, and Jun- jie Yan. 2022. Supervision exists everywhere: A data efficient contrastive language-ima image pre-training paradigm. In ICLR. OpenReview.net.
Henri A Favre and Warren H Powell, 2013. Nomencla- ture of organic chemistry: IUPAC recommendattons and preferred names 2013, Royal Society of Chem- istry.
Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulci Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu. Wei Lin, Jin- rui Yang. Xiawu Zheng, et al. 2023. Mme: A compre- hensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306. 10..306633344 13394.
Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mo- hta, Tenghao Huang, Mohit Bansal, and Colin Raf- fel. 2022a. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. In NewrIPS.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language under- standing. In ICLR. OpenReview.net.
Haotian Liu, Chumyuan Li, Qingyang Wu, and Yong Jac Lee. 2023. Visual instruction tuning, arXiv preprint 1xXxi:2334.08445
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang. and Weizhu Chen. 2022. Lora: Low-rank adaptation of large language models. ICLR. OpenReview.net.
Shengchao Liu, Weili Nie, Chengpeng Wang. Jiarui Lu, Zhuoran Qiao, Ling Liu, Jian Tang. Chaowei Xiao, and Anima Anandkumar, 2022b. Multi-modal molecule structure-text model for text-based retrieval and editing. CoRR, abs/2212. 10789.
Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitmik, Percy Liang. Vijay Pande, and Jure Leskovec. 2020. Strategies for pre-training graph neural networks. In ICLR.
Ilya Loshchilov and Frank Hutter, 2019, Decoupled weight decay regularization. In ICLR (Poster). Open- Review.net.
Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W. Cohen, and Xinghua Lu. 2019. Pubmedqa: A dataset for biomedical research question answering. In EMNLP/IJCNPP (1), pages 2567-2577. Association for Computational Linguistics.
Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, and Sayak Paul. 2022. Peft: State- of-the-art parameter-efficient fine-tuning methods. http://ggttub.cccco http:///ggttuu..oo//ugggggg
Nitish Shirish Keskar, Bryan McCann, Lay Varshney, Caiming Xiong. and Richard Socher. 2019. Ctrl: A conditional transformer language model for control- lable generation. arXiv preprint arXiv:1909.000 005588.
Jack Merullo, Louis Castricato, Carsten Eickhoff, and Ellie Pavlick. 2023. Linearly mapping from image to text space. In JCLR.
Sunghwan Kim, Jie Chen, Tiejun Cheng, Asta Gindu- lyte, Jia He, Siqian He, Qingliang Li. Benjamin A. Shoemaker, Paul A. Thiessen, Bo Yu. Leonid Za- slavsky, Jian Zhang, and Evan Bolton, 2021. Pub- chem in 2021: new data content and improved web interfaces. Nucleic Acids Res., 49tDabbas I338e):
OpenAl. 2023, GPT-4 technical report. abs/2303.08774.
CORR.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: method for automatic evalu- ation of machine translation. In ACL, pages 311-318. ACL.

---

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learn- ing transferable visual models from natural language supervision. In ICML,, volume 139 of Proceedings of Machine Learning Research, pages 8748-8763. PMLR.
David Weininger. 1988. Smiles, a chemical language and information system. 1. introduction to methodol- ogy and encoding rules. J. Chem. Inf. Comput. Sci., 1-:33333
Alexander Frank Wells. 2012. Structural inorganic chemistry. Oxford university press.
Zhenqin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh S Pappu, Karl Leswing, and Vijay Pande. 2018. Moleculenet: benchmark for molecular machine learning. Chem- ical science, 92)):513-530
Colin Raffel, Noam Shazeer. Adam Roberts. Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. J. Mach. Learn. Res., 1-140:67.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanic Jegelka. 2019. How powerful are graph neural net- works? In ICI.R.
Yu Rong, Yatao Bian, Tingyang Xu, Weiyang Xic, Ying Wei, Wenbing Huang, and Junzhou Huang. 2020. Self-supervised graph transformer large- scale molecular data. In NeurlPS.
Lewei Yao, Runhui Huang. Lu Hou, Guansong Lu. Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. 2022. FILIP: fine- grained interactive language-iman pre-training, In ICLR. OpenReview.net.
Philipp Seidl, Andreu Vall, Sepp Hochreiter, and Günter Klambauer. 2023. Enhancing activity pre- diction models drug discovery with the ability to understand human language. arXiv preprint arXiv:2303.03363.
Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. 2020. Graph con- trastive learning with augmentations. In NeurIPS.
Teague Sterling and John J. Irwin. 2015. ZINC ligand discovery for everyone. J. Chem. Inf. Model., (5(11):2324-233)
Zheni Zeng, Yuan Yao, Zhiyuan Liu, and Maosong Sun. 2022. A deep-learning system bridging molecule structure and biomedical text with comprehension comparable to human professionals. Nature commu- nications, 13(1):862.
Bing Su. Dazhao Du. Zhao Yang, Yujie Zhou, Jiang- meng Li, Anyi Rao, Hao Sun, Zhiwu Lu, and Ji- Rong Wen. 2022. A molecular multimodal founda- tion model associating molecule graphs with natural language. CoRR, Abs/2209.0548)
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona T. Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Oit, Sam Shleifer, Kurt Shus- ter, Daniel Simig. Punit Singh Koura, Anjali Srid- har. Tianlu Wang, and Luke Zettlemoyer. 2022. OPT: open pre-trained transformer language mod- cls. CORR, abs/2205.0106 00006...
Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, An- drew Poulton, Viktor Kerkez, and Robert Stojnic. 2022. Galactica: A large language model for science. CoRR, abs/2211 111111.000 1.09885.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux. Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. CoRR, 11t/2322..1991...
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang. Xiaolei Wang, Yupeng Hou, Yinggian Min, Be- ichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang. Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu. Peiyu Liu, Jian Yun Nie, and Ji-Rong Wen. 2023. A survey of large language models. CoRR. Ahs/2303.18222
Maria Tsimpoukelli. Jacob Menick, Serkan Cabi, S. M. Ali Eslami, Oriol Vinyals, and Felix Hill. 2021. Multimodal few-shot learning with frozen language models. In NeuriPS, pages 200-212.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. Minigpt-4: Enhancing lasion-language understanding with advanced large language models. arXiv preprint 2304 10592.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS, pages 5998 6008.
Alain Vaucher, Philippe Schwaller, Joppe Geluykens, Vishnu H Nair, Anna Iuliano, and Teodoro Laino. 2021. Inferring experimental procedures from text- based representations of chemical reactions. Nature communications, 12(1):3573.

---

A Complete Related Works
Vision-Language Pretraining (VLP). Both VLP and Molecular Language Modeling aim bridge the gap between text and another modality. Notably, VLP methods of CLIP (Radford et al., 2021) and others (Li et al., 2022; Yao et al., 2022) use contrastive learning to connect a visual encoder and a text encoder. These methods can be applied for tasks like image-text retrieval and zero-shot image classification. Recently, series of VLP works (Tsimpoukelli et al., 2021; Merullo et al., 2023; Li al., 2023; Alayrac et al., 2022) show that visual features can be aligned to the text space of LMs. This cross moosl alignment allows LMs to utilize their language generation and few-shot learn- ing abilities for multi-modal tasks. MolCA draws inspiration from these findings. To the best of our knowledge,
We present the complete literature review. In addi- tion to the molecule-related literature, as addressed in the main body of the paper, we also discuss MolCA's relation to vision-language pretraining.
Molecule Understanding via 1D Language Modeling. Due to the extensive biochemical liter- ature in their training corpus, some open-domain LMs (Zhang et al., 2022: Touvron et al., 2023; Chowdhery et al., 2022) have obtained a high-level understanding of molecular and chemical concepts. This is demonstrated through their promising per- formances in text-related biochemical and medical question-answering benchmarks (Hendrycks et al., 2021; Jin al., 2019). Among these LMs, Galac- tica (Taylor al., 2022) shows competitive perfor- mances for using corpus that primarily com- posed of scientific literature. Focusing on the chem- istry domain, KV-PLM (Zeng et al., 2022) models molecules by applying masked language modeling loss on ID SMILES. Vaucher et al. (2021) pro- pose to predict the chemistry experiment actions by reading chemical reaction equations. MolT5 (Ed- wards et al., 2022) presents several T5-based (Raf- fel al., 2020) LMs for SMILES-to-text and text- to-SMILES translations. Further, Christofidellis et al. (2023) propose to fine-tune T5 for chem- ical reaction prediction and retrosynthesis tasks. MolCA is different from these methods that exclu- sively utilize ID SMILES to represent molecules. Instead, MolCA aims to enable LMs to perceive molecules' 2D graph representations.
the first to align 2D molecu- lar graphs to the text space of LMs. Furthermore. we incorporate uni-modal adapter to improve the adaptation efficiency on downstream tasks.
B Experimental Settings
Pretrain Settings. MolCA's pretrain stage has 50 epochs and pretrain stage 2 has 10 epochs. Q- Former has 8 query tokens (N, = 8). Our opti- mizer's configuration follows (Li et al., 2023), We use the A optimizer (Loshchilov and Hut- ter, 2019) with a weight-decay of 0.05. The learn- ing rate is scheduled by combination of linear warmup and cosine decay. The peak learning rate is le-4 and the warmup has 1000 steps.
Molecule Captioning. MolCA is fine-tuned for 100 epochs using the same configuration of optimizer and learning rate scheduler. LoRA is implemented using the OpenDelta library (Ding et al., 2022) and the PEFT library (Mangrulkar et al., 2022). For the PubChem324k dataset, we set LoRA's rank to 8 and apply LoRA to Galactica's 's modules of [q_proj, v_proj]. This configura- tion yields a LoRA adapter with 2M parameters, which constitutes 0. of the parameters the Galacticai,sa For the CheBI-20 dataset, we set LoRA's rank r to 16 and apply LoRA to Galactica's modules of [q_proj, v_proj, out_proj, fc1, fc2]. This configuration yields LoRA adapter with 12M parameters, which constitutes 0.94% of the parameters in the Galacticar.sn IUPAC Name Prediction. We collect TUPAC names for molecules in the train/valid/test sets of PubChem324k using the PubChemPy library?. The
Molecule- Text Contrastive Learning. Driven by the demand of a molecule-text retrieval system, Tex12Mol (Edwards et al., 2021) employs cross- modal contrastive learning to train a molecular graph encoder of GCNs (Kipf and Welling, 2017) and a text encoder of Sci-BERT (Beltagy al., 2019). Subsequent works (Su et al., 2022; Liu et al., 2022b; Seidl et al., 2023) have proposed im- provements, including the addition of inter-modal contrastive learning loss (Su et al., 2022) and apply- ing the model for text-based molecule editing (Liu et al., 2022b). However, moosl contrastive learning is unsuitable for open-ended conditional generation task (Alayrac al., 2022), because of its focus on learning similarity function. To resolve the problem, we propose MolCA enable the LM's understanding of 2D molecular graphs, facili- tating MolCA's capability of open-ender molecule- to-text generation.
Pht //gi thub. com wom/mcs07//Pee/ /PubChemPy

---

Subset
#Mol-text pairs
Usage
Avg mol
Avg text len
Min text len
Max text
Pretrain
309689
Pretrain stage &
35
18
1
1305
Train 12000
Downstream fine-tune
32
60
20
937
Valid
1000
Downstream validation
32
61
20
1197
Test
2000
Downstream
31
60
20
879
Table Statistics of the PubChem324k dataset.
Retrieval in batch
Retrieval in test set
M2T (%)
T2M (%)
M2T (%)
T2M (%)
Model
Ace
R@20
Acc
R@20
Acc
R@20
R@20
1D SMILES
Sci-BERT
84.1
98.6
82.5
98.2
39.3
86.1
37.9
85.1
KV-PLM
84.3
98.3
82.3
98.3
38.8
86.3
38.7
85.6
2D Graph
MoMu-S*
42.3
90.1
43.7
90.1
11.5
41.2
12.6
43.6
MoMu-K*
43.3
90.4
45.8
89.0
11.3
41.0
12.4
39.9
MoMu-S
82.9
99.0
83.0
99.0
40.6
86.5
40.6
86.5
MoMu-K
84.1
98.7
83.6
98.9
41.8
88.9
42.4
88.5
MoleculeSTM
87.2
98.5
86.7
98.3
47.1
89.0
45.4
91.5
MolCA w/o MTM
86.6
98.9
85.3
98.7
60.5
93.7
58.6
92.3
MolCA
91.4
99.9
90.1
99.2
69.4
95.7
69.6
94.6
(a) Molcule-text retrieval performances in the PubChem324k dataset.
Retrieval batch
Retrieval in test set
M2T (%)
T2M (%)
M2T (%)
T2M (%)
Model
Acc
R@20
Acc
R@20
Acc
R@200
Acc
R@20
1D SMILES
Sci-BERT!
62.6
61.8
60.7
60.8
KV-PLM*
77.9
65.0
75.9
64.3
2D Graph
MoMu-St
80.6
77.0
79.1
75.5
MoMu-Kt
81.1
80.2
80.2
79.0
MoleculeSTM
86.2
83.9
84.6
85.1
MolCA wo MTM
88.4
98.8
85.5
98.5
54.6
88.0
51.8
85.5
MolCA
91.4
99.8
88.4
98.8
60.3
90.5
59.7
87.6
(b) Molecule-text retrieval performances in the PCDes dataset.
Retricval hatch
Retrieval in
M2T
T2M
M2T (%)
T2M (%)
Model
Acc
R
Ace
R@20
Acc
R@20
Ace
6
1D SMILES
Sci-BERT
1.4
1.6
0.3
0.3
KV-PLM
1.5
1.3
0.3
2D Graph
MoMu-S!
45.7
40.0
43.3
43.4
MoMu-Kt
46.2
38.5
43.7
43.5
MoleculeSTM
81.8
81.9
75.8
74.5
MolCA w/o MTM
77.3
97.7
77.3
97.5
38.1
81.5
37.3
81.6
MoICA
83.7
98.9
84.3
98.7
48.4
88.6
48.3
87.3
(c) Molecule-text retrieval performances in the MoMu dataset.
Table 7: Complete molecule-text retrieval performances the datasets of PubChem324k, PCDes and MoMu. denotes performance evaluated on the baseline's 's released checkpoint. i denotes result borrowed from (Su et al., 2022). Other models are trained on PubChem324k pretrain subset.
experiment uses the same hyperparameters as the molecule captioning experiment. We append a text
prompt "The molecule's IUPAC name is" after the molecule representations as the task description (cf.

---

Model
Pretrain stage
Pretrain stage 2 BLEU-2
BLEU-4
ROUGE-1
ROUGE-2 ROUGE-L METEOR
MolCA, Galaci.se
K
35.8
27.6
47.4
33.0
42.1
42.2
MolCA, Galaci.s
X
37.7
29.5
49.2
34.9
43.8
44.5
MolCA, Galaci.s
V
39.8
31.7
51.7
37.3
46.2
46.8
Table 8: Ablating MolCA's two pretrain stages by the task of molecule captioning in the PubChem324k dataset.
Cross-Modal Projector Representation Type
BLEU-2 BLEU-4
ROUGE-1
ROUGE-2
ROUGE-L
METEOR
ID SMILES
33.7
26.0
45,4
31.6
40.7
40.3
Linear Q-Former
ID SMILES Graph
34.6
46.7
32.9
41.2
ID SMILES Graph
39.8
31.7
37.3
46.2
Table 9: Comparing different cross-modal projectors for molecule captioning on the PubChem324k dataset. All the compared methods apply LoRA fine-tuning on Galactica
Molecule
Ground truth
MolCA
Molecule
Ground truth
MoiCA
128 A-daminnodd-
(25)-2 aamindd-
joined
dycooiicc
juined
by
bond is dleaccharde derkotive, aigesaoch anide sultiee member of sultamic aolss.
-4] glycosidis bond. E disacchario.. an
amino araccharcee,, and member of sulfomii
monooattecsyl
cihoow 2,5 beiien
The mo is long tongechen issty aeat that is behenic aci susetiured cost on by hydroay
The molezzil is d-tyytrocy fiety acad that
inpomorpha adld
resatosannic add cubsiiluted at postion
carbogyl
related hydroxy faity functionally a hhydhcagteheriins.
Is
functionally
group. long
and
2egl--di xylocol add
conjugate and o 2tydhcagheaaceenne
nonylaoedi and
(a) Samples of molecule captioning.
(b) Samples of IUPAC name prediction.
Figure 7: Examples of MolCA's molecule-to-tee generation results. We highlight text snippets blue that correctly describe the molecule structures in the predicted texts. To save space, some parts of texts are replaced by (...).
Figure 5).
including optimizer and LoRA, are the same as the Molecule Property Prediction experiment.
Molecule-Texx Retrieval. We use MolCA's checkpoint from pretrain stage I for retrieval with- out fine-tuning on any other datasets. This is simi- lar to the setting of zero-shot retrieval in (Su et al., 2022; Liu et al., 2022b).
Galactica. Following the instructions in (Tay- lor et al., 2022), we wrap SMILES sequences with special tokens of [START_I_SMILE [END_I_SMILES] before feeding them into Galac- tica.
and
Molecule Property Prediction. Following (Hu et al., 2020), we fine-tune the models for 100 epochs and report the test performance selected by the valid set. For molecule classification, we attach a linear classifier after the mean pooling of the LM's hidden states of the last layer. We use the AdamW optimizer with a constant learn- ing rate of le-4 and weight decay of 0.05. This experiment uses the same LoRA configuration as the molecule captioning experiment the Pub- Chem324k dataset.
PubChem324k Dataset. Our dataset collection process follows the procedures described in (Liu et al., 2022b). The resulting dataset is larger due to the frequent updates made to the PubChem database (Kim et al., 2021). For each molecule in this website, we use the "description" field its webpage as the corresponding text description. To avoid information leakage, we replace any com- mon name or IUPAC name of the molecule at the beginning of texts with text template (i.e., "The molecule"). Detailed statistics of PubChem324k presented in Table 6.
Counting Functional Groups (FGs). We use the molecules in PubChem324k 's train set for fine- tuming and use the molecules in the valid set for evaluation. Following (Rong et al., 2020), we RDkit (Landrum, 2013) to obtain the ground truth counts of FGs in every molecule. For each FG type. we employ a separate linear classifier to regress its numbers. Our model is trained using the Mean Square Error (MSE) loss function. Other settings,
C More Experimental Results
Molecule-Text Retrieval. Here we present MolCA's complete molecule-text retrieval perfor- mance on the PuhChem324k, PCDes, and MoMu datasets. Following (Su et al., 2022), we report the performance of retrieval in a batch of 64 random

---

Sample 1
SMILES: Ciic@ @icceeeeee CCOEENIIIITTTTTTEEEEEEIHOOOOOOOO [C@ (C@CHICC@HCCCCCCCC wtt.CCCCCCCCCCCCCCCC/ooOOOOOOS.D
Ground truth
The molecule is an amino disaccharide consisting of alpha-(...) joined in sequence by (1->4) glycosidic bond. It is disaccharide derivative, oligosaccharide sulfate, member of sulfamic acids, a monocarboxylic acid (...)
ID SMILES
The molecule is disaccharide sulfate consisting of 2-acetamido-(..) joined in sequence by (1-24) glycosidic bond. It is functionally related to N-acetyl-D-glucosee and a N-acetyl-D- galactosamine.
ID SMILES + 2D Graph
The molecule is a disaccharide that consists of 2-O-(...) residues joined sequence by a (1 (1-24) glycosidic bond. It is a disaccharide, an amino disaccharide, and a member of sulfamic acids.
Sample 2
SMILES: WC
Ground truth
The molecule is long-chain fatty acid that is hehenic acid substitured at position by hydroxy group. It is 2- 2-hydroxy fatty acid. It is functionally related to docosanoic acid. It is conjugate acid of 2-hyyrroyychhentee.
ID SMILES
The molccule is a 2-hydroxy fatty acid that is the 2-hydroxy derivative of tetracosanoic acid. It is functionally related to a tetracosanoic acid. It is conjugate acid of a 2- hyydooyyetrrccosanoate
ID SMILESS 2D Graph
The molecule is 2-hydroxy fatty acid that is hexacosanoic acid substituted at position by a hydroxy group. It is long-chain fatty acid. is functionally related to an hexacosanoic acid. It a conjugate acid a --hyyroxyhexcoosamoee
Table 10: Molecule captioning samples of MolCA (i.e.. ID SMILES 2D Graph) and its variant of using only ID SMILES. We highlight text snippets in blue that correctly describe the molccule structures the predicted texts. To save space, some parts of texts replaced by (...).
samples and the performance of retrieval in the en-
(Zhu et al., 2023)) outperform linear cross- projector based method (e.g.. LLaVA (Liu et al., 2023)). 2) Linear cross-modal projector slightly outperforms the SMILES-only baseline. We at- tribute this improvement to the usage of 2D molec- ular graphs, but the gains are limited because the linear projector is less effective.
tire test set. As shown in Table 7, our conclusions align with those from Section 4.4: 1) MolCA con- sistently outperforms the baselines for molecule- text retrieval; 2) applying the MTM module for re-ranking is crucial for MolCA's molecule-text retrieval performances.
MolCA's Generation Results. Figure 7 shows MolCA's molecule-to-terr generation results. The two samples of molccule captioning is also pre- sented in Table 10. Specifically, we compare MolCA (Lee. ID SMILES + 2D Graph) and its variant that is pretrained and fine-tuned using only ID SMILES. We can observe that using both ID SMILES and 2D graph leads more accurate de- scriptions of molecule structures.
Ablating the Pretrain Stages. We conduct ab- lation studies on MolCA's two pretrain stages. As shown in Table 8, both the two pretrain stages have significant contributions to MolCA's molecule cap- tioning performances.
Ablating the Cross-Modal Projector. We com- pare the performances of our selected cross-modal projector Q-Former and a linear cross-modal pro- jector. For the linear cross-modal projector, feed the node representations from the graph en- coder to the base LM after the linear projector layer. We tune the weights of the graph encoder, linear projector, and the base LM's LoRA adapter. The experimental setting and hyperparameters are the same as those of MolCA. Table shows the results. We can observe that: I) Linear cross-modal projec- tor underperforms Q-Former. We conjecture that linear layer suboptimal to bridge the modal- ity gap between 2D molecules and ID texts. This aligns with findings in the MME benchmark (Fu et al., 2023), where Q-Former- BLIP-2, InstructBLIP (Dai et al., 2023), MiniGPT.
Computational Cost. We present the real-world training time of MolCA's three training stages Table 11. All experiments are conducted on two NVIDIA A100 40 GB GPUs. Notably, we observe that the fine-tuning stage is affordable in terms of computational resources.
methods (e.g.,

---

Stage
LM
Dataset
Epochs
Time
Pretrain stage
PubChem324k pretrain subset
50
18,0h
Pretrain stage
Galacııa, freeze
PuhChem324k pretrain subset
10
9.0h
Pretrain stage
Galacraa, freeze
PubChem324k pretrain subset
10
3.0h
Fine-tune stage
Galacy,u, LoRA
PuhChem324k train subset
100
6.0h
Fine-tune stage
Galacraa, full
PubChem324k train subset
100
1.5h
Table 11: Compuational cost for MolCA's three stages.


Reviews:
[{"review_id": "aMek4F2xfd", "reviewer": "Reviewer_YfAf", "summary": "", "questions": "How could we improve the Q-former to make MolCA sufficient for practical application? In addition to collect a larger pre-training corpus, do we need other tasks besides molecular captioning to align the representations of 2D molecular structure to text representation space?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper. We're genuinely pleased to hear that you found the approach novel and the performance superior. The constructive feedback provided will undoubtedly help us further refine and improve our work. Here we provide a detailed response to each of your suggestions.\n\n> **Q1:** Although the author ablate MolCA’s capability of counting FGs types inside molecules to demonstrate the LMs could understand 2D molecular structure, there is a lack of further in-depth analysis of how well the Q-former can attend to the corresponding graph structure conditioning on the textual input. For example, when the model caption a specific function of the given molecule, does the corresponding graph nodes have a high attention value?\n\n**Response:** Thank you for your insightful comment. We appreciate your suggestion, but would like to clarify the inherent technical challenges in assessing the node-text attention map. To elucidate, let's consider two distinct stages:\n\n* **Pretrain Stage 1.** Q-Former establishes attentional relationships between graph nodes and query tokens (i.e., node-query attention map), as well as between query tokens and texts (i.e., text-query attention map). However, there isn't a direct attention connection between graph nodes and texts, thereby precluding direct access to the node-text attention map.\n\n*  **Pretrain Stage 2 and Fine-tune Stage.** During these stages, Q-Former processes graph node inputs, without the text inputs. As a result, the graph nodes lack the capacity to attend to texts within Q-Former's framework. Although the base LM has text inputs, its 2D graph inputs are not provided on a node-wise basis, making it difficult to evaluate attention values between texts and nodes.\n\nWe fully recognize the importance of node-text attention map and will explore it in future work, to resolve the outlined challenges.\n\n\n\n> **Q2:** How could we improve the Q-former to make MolCA sufficient for practical application? In addition to collect a larger pre-training corpus, do we need other tasks besides molecular captioning to align the representations of 2D molecular structure to text representation space?\n\n**Response:** Thanks for the insightful question on improving practical application performance. As you rightly pointed out, collecting a larger pre-training corpus is a straightforward method for performance improvement. And we agree that a larger dataset alone may not be sufficient. Beyond that, we foresee two other potential methods for improvement:\n\n* **Molecular Graph-Language Modeling with Multiple Molecules.** Simultaneously modeling multiple molecules requires the LM to understand and process the interactions and relations between multiple molecule inputs, an ability that can hardly be learned by modeling a single molecule. Properly cultivating this ability can potentially benefit downstream tasks that require reasoning across multiple molecules, such as chemical reaction prediction [1] and retrosynthesis [2]. \n\n* **Molecule-centric Instruction Tuning.** It is possible to improve MolCA's performance by aligning its generation results to human preference using instruction tuning [3,4,5]. We speculate that molecule-centric instruction tuning can be achieved by fine-tuning MolCA on a diverse set of molecule-related text generation tasks. It can potentially improve MolCA in two distinct ways:\n  * It augments MolCA's ability to interact with humans. For example, if molecule-centric question-answering data can be included in instruction tuning, we can let MolCA's generation results focus on specific molecule properties by raising the corresponding question, rather than merely presenting all relevant information.\n  * If chain-of-thought annotation data can be included, it can potentially improve MolCA's ability of few-shot learning [3,4]. For pure texts, chain-of-thought annotations decompose each task into a sequence of sub-tasks [6]. Through learning to perform these sub-tasks, LMs have potentials to generalize to new tasks using a few examples [6].\n\n\n\n> **Q3:** There are some works focusing on Graph-Aware Language Model Pre-Training, e.g., Xie, Han, Da Zheng, Jun Ma, Houyu Zhang, Vassilis N. Ioannidis, Xiang Song, Qing Ping et al. \"Graph-Aware Language Model Pre-Training on a Large Graph Corpus Can Help Multiple Graph Applications.\" arXiv preprint arXiv:2306.02592 (2023).\n\n**Response:** Thank you for kindly pointing out the concurrent reference by (Xie et al., 2023). Indeed, the work is relevant as it also focuses on joint learning of 2D graphs and 1D texts. We did not include this concurrent work in our initial submission due to its public availability on June 5, 2023, which fell within the same month of EMNLP's submission deadline. We have promptly addressed this oversight by incorporating this work into our revised related work section.\n\nIn comparison with the referenced work, we identify two key distinctions that set our work apart:\n\n* **Different Domains.** We focus on molecular graphs while the reference focuses on social networks. The graphs from different domains differ in both structure and scale, which calls for different technical solutions.\n\n* **Different Purposes.** We aim to enable an LM to understand 2D molecular graphs. In contrast, the referenced work leverages the LM as a text feature extractor for Graph Neural Networks.\n\nWe believe these clarifications highlight the uniqueness of our approach, and we appreciate the opportunity to enhance our paper with the inclusion of this relevant reference.\n\n\n\n**Reference:**\n\n[1] Chemical-reaction-aware molecule representation learning. In ICLR 2022\n\n[2] Retrosynthetic reaction pathway prediction through neural machine translation of atomic environments. In Nature Communications 2022.\n\n[3] The Flan Collection: Designing Data and Methods for Eﬀective Instruction Tuning. In Arxiv 2023.\n\n[4] Scaling Instruction-Finetuned Language Models. In Arxiv 2022.\n\n[5] InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning. In Arxiv 2023.\n\n[6] Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. In NeurIPS 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Existing work exclusively utilizes 1D SMILES to represent molecules, while this paper proposed a a novel molecular language modeling method, which enable LMs to perceive molecules’ 2D graph representations. The model features a cross-modal Q-former to map representations of 2D graphs into the text space of LMs and applies an adapter for efficient downstream-finetuning. \n\nThe author evaluate the proposed method for molecule captioning and IUPAC name prediction  task on PubChem324k and CheBI-20, demonstrating a consistent superiority to the baseline methods.  Additionally, the proposed method also outperform the existing baseline on the Molecule-Text Retrieval task on three benchmark datasets. The author also conducts ablation studies to demonstrate that combing 2D graphs and 1D SMILES leads to improved performance.", "reasons_to_reject": "Although the author ablate MolCA’s capability of counting FGs types inside molecules to demonstrate the LMs could understand 2D molecular structure, there is a lack of further in-depth analysis of how well the Q-former can attend to the corresponding graph structure conditioning on the textual input. For example, when the model caption a specific function of the given molecule, does the corresponding graph nodes have a high attention value?", "questions_to_authors": "How could we improve the Q-former to make MolCA sufficient for practical application? In addition to collect a larger pre-training corpus, do we need other tasks besides molecular captioning to align the representations of 2D molecular structure to text representation space?", "paper_topic_and_main_contributions": "The paper studies the open-ended molecule-to-text generation problem. The key point is to let language models understand the 2-dimensional molecule graph and translating the representations of 2D graphs into 1D soft prompts in the text space, and thus, bridging the gap between the graph encoder’s representation space and the LM’s input space. To this end, the author proposed MolCA and applies a three-stage training pipeline. In the first stage, a cross-modal projector and the graph-encoder are trained to extract the relevant molecule features to given the text, endowing the model with powerful molecule-text retrieval ability. In the second stage, the cross-modal projector is connected to a frozen LM and learn to produce soft prompts that the LM can understand. In the third stage, MolCA is fine-tuned for downstream generation tasks. Empirical results demonstrate that MolCA achieves SoTA results on various tasks, including molecule captioning and molecule-text retrieval.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "There are some work focusing on Graph-Aware Language Model Pre-Training, e.g., Xie, Han, Da Zheng, Jun Ma, Houyu Zhang, Vassilis N. Ioannidis, Xiang Song, Qing Ping et al. \"Graph-Aware Language Model Pre-Training on a Large Graph Corpus Can Help Multiple Graph Applications.\" arXiv preprint arXiv:2306.02592 (2023)."}}, {"review_id": "jCc56DJZRL", "reviewer": "Reviewer_JBDY", "summary": "", "questions": "1. For training two Galactica models in your framework, what is the training cost (e.g., cost of time / gpus) for completing both pretrain stage 1 and stage 2?\n2. Does MolCA has potential to extend to molecule classification and generation tasks? I notice in Table 5, although adding topological information, the improvement of performance is still limited, and is worse than some pretrained molecule encoder, which is trained only on 2D molecular graph data.\n3. Can you provide results of MolT5 + MolCA and Galac with 1D smiles only?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the time you've invested in reviewing our paper and are gratified to know that you consider our idea novel and has great potential. Your constructive insights are invaluable for the ongoing improvement of our research. Below, we address each of your comments in detail.\n\n> **Q1:** It seems that the authors did not reveal the computational cost of training Galactica with MolCA's training pipeline, this hinder the understanding of the price in order to gain improvement claimed in the paper. ... For training two Galactica models in your framework, what is the training cost (e.g., cost of time / gpus) for completing both pretrain stage 1 and stage 2?\n\n**Response:** Thank you for the advice on detailing the computational cost. We have revised our submission to report the computational costs for both pretraining and fine-tuning stages. The results are shown in the Table 1 below. Specifically, all experiments are conducted on two NVIDIA A100 40 GB GPUs. \n\nNotably, we observe the fine-tuning stage to be affordable in terms of computational resources. To further aid in the adoption of our method, we plan to release our pretrained models for public use.\n\n**Table 1:** Compuational cost for MolCA's three stages.\n\n| Stage            | Base LM                        | Dataset                     | Epochs | Time  |\n| ---------------- | ------------------------------ | --------------------------- | ------ | ----- |\n| Pretrain stage 1 | -                              | PubChem324k pretrain subset | 50     | 18.0h |\n| Pretrain stage 2 | Galac$_{\\text{1.3B}}$, freeze  | PubChem324k pretrain subset | 10     | 9.0h  |\n| Pretrain stage 2 | Galac$_{\\text{125M}}$, freeze  | PubChem324k pretrain subset | 10     | 3.0h  |\n| Fine-tune stage  | Galac$_{\\text{1.3B}}$, LoRA ft | PubChem324k train subset    | 100    | 6.0h  |\n| Fine-tune stage  | Galac$_{\\text{125M}}$, full ft | PubChem324k train subset    | 100    | 1.5h  |\n\n> **Q2:** In their experiments, the authors only report results for MolT5 in 1D modeling and MolCA + Galac in 1D+2D modeling, making direct comparisons difficult. ... Can you provide results of MolT5 + MolCA and Galac with 1D smiles only?\n\n**Response:** Thanks for the advice. We agree that these comparisons are crucial for a comprehensive understanding of MolCA's performance. For this purpose, we report performances of MolCA's variant that uses 1D SMILES only (*i.e.*, LoRA fine-tuned Galactica1.3B), and MolCA with different base LMs (*i.e.*, MolCA's MolT5 version).\n\n**Comparison with LoRA fine-tuned Galactica1.3B.** We have updated Table 5a in our submission to include Galactica1.3B's performance on the CheBI-20 dataset alongside the existing results on the PubChem324k dataset. Table 5a is shown below. We observe that adding 2D graph representations can significantly improve the performances for molecule captioning and IUPAC name prediction, which is consistent with the existing observations.\n\n**Table 5a:** Ablating the representation type on the PubChem324k dataset. All compared models fine-tune the base LM of Galactica1.3B. \n\n| Representation type                     | BLEU-2   | BLEU-4   | ROUGE-1  | ROUGE-2  | ROUGE-L  | METEOR   |\n| --------------------------------------- | -------- | -------- | -------- | -------- | -------- | -------- |\n| **Molecule  Caption, PubChem324k**      |          |          |          |          |          |          |\n| SMILES                                  | 33.7     | 26.0     | 45.4     | 31.6     | 40.7     | 40.3     |\n| Graph                                   | 35.7     | 27.4     | 47.3     | 32.3     | 41.8     | 42.0     |\n| SMILES+Graph                            | **39.8** | **31.7** | **51.7** | **37.3** | **46.2** | **46.8** |\n| **Molecule  Caption, CheBI-20**         |          |          |          |          |          |          |\n| SMILES                                  | 58.3     | 49.4     | 65.9     | 51.3     | 59.7     | 62.4     |\n| SMILES+Graph                            | **63.9** | **55.5** | **69.7** | **55.8** | **63.6** | **66.9** |\n| **IUPAC  name Prediction, PubChem324k** |          |          |          |          |          |          |\n| SMILES                                  | 70.7     | 60.7     | 68.6     | 46.2     | 61.7     | 71.5     |\n| SMILES+Graph                            | **74.6** | **66.1** | **70.5** | **49.1** | **64.2** | **73.0** |\n\n**Comparison with MolCA's MolT5 version.** We have updated Table 2a in our submission to include the performance of MolCA using MolT5-large as the base LM. We can observe that MolCA with MolT5-large consistently outperforms MolT5-large alone, showing MolCA's effectiveness in enabling LM to perceive 2D graphs.\n\n**Table 2a:** Performances (%) of molecule captioning on the PubChem324k datasets.\n\n| Model                     | BLEU-2   | BLEU-4   | ROUGE-1  | ROUGE-2  | ROUGE-L  | METEOR   |\n| ------------------------- | -------- | -------- | -------- | -------- | -------- | -------- |\n| **1D SMILES**             |          |          |          |          |          |          |\n| MolT5-small               | 14.8     | 8.5      | 26.5     | 13.5     | 23.6     | 18.5     |\n| MolT5-base                | 30.1     | 20.9     | 40.3     | 25.1     | 33.8     | 35.6     |\n| MolT5-large               | 30.2     | 22.2     | 41.5     | 25.9     | 34.8     | 36.6     |\n| **1D  SMILES + 2D Graph** |          |          |          |          |          |          |\n| MoMu-small                | 19.1     | 12.0     | 29.7     | 16.3     | 26.7     | 21.8     |\n| MoMu-base                 | 30.2     | 21.5     | 40.5     | 25.1     | 34.4     | 34.2     |\n| MoMu-large                | 31.1     | 22.8     | 41.8     | 25.7     | 36.7     | 36.2     |\n| MolCA, MolT5-large        | 33.7     | 27.0     | 49.7     | 35.6     | 44.4     | 42.4     |\n| MolCA, Galac125M          | 32.4     | 24.9     | 44.9     | 30.1     | 39.5     | 39.2     |\n| MolCA, Galac1.3B          | **39.8** | **31.7** | **51.7** | **37.3** | **46.2** | **46.8** |\n\nYour suggestions have been invaluable in refining our paper, and we believe these updates will provide a more comprehensive evaluation of MolCA's capabilities and performances.\n\n> **Q3:** Does MolCA has potential to extend to molecule classification and generation tasks? I notice in Table 5, although adding topological information, the improvement of performance is still limited, and is worse than some pretrained molecule encoder, which is trained only on 2D molecular graph data.\n\n**Response:** Thank you for your insightful question and the observation of MolCA's potential in the molecule classification task. In brief, MolCA does have the potential to be extended to molecule classification, and MolCA's base LM, Galactica, can be adapted for molecule generation, following [1]. \n\nNow we elaborate in detail the possible methods to fully unleash the potential of MolCA and Galactica for molecule classification and molecule generation.\n\n* **Molecule Classification.** Compared to pretrained 2D molecular graph encoders, MolCA's advantage is the usage of an LM. Therefore, LM's abilities like in-context learning and chain-of-thought reasoning [2,3,4] can potentially improve molecule classification performance, especially in the few-shot setting and out-of-distribution setting. To fully unleash these abilities, we speculate that it is necessary to first fine-tune MolCA with molecule-centric instruction tuning data, which has been discussed in our response to Q2 of Reviewer YfAf. \n* **Molecule Generation.** MolCA's base LM, Galactica, can be fine-tuned for this task, following [1]. To achieve the optimal performance, we suggest first continuing pretrain Galactica on a large collection of molecule sequences, or weakly supervised (molecule, text) pairs, before applying it for text-based molecule generation. \n\nRegarding the performance of molecule classification, the current MolCA model is limited by the adopted molecular graph encoder and the molecule features. The most recent molecular graph encoders rely on molecules' 3D coordinates and carefully designed fingerprint features [5,6]. However, considering that molecule classification is not the main focus/contribution of this work, we leave the adoption of these advanced molecular graph encoders and molecule features to future exploration. \n\nIn summary, we view these extensions as promising future directions. MolCA's ability to interpret the knowledge in text corpus provides it an advantage over uni-modal molecular methods.\n\n\n\n**Reference:**\n\n[1] Translation between Molecules and Natural Language. In EMNLP 2022.\n\n[2] The Flan Collection: Designing Data and Methods for Eﬀective Instruction Tuning. In Arxiv 2023.\n\n[3] Scaling Instruction-Finetuned Language Models. In Arxiv 2022.\n\n[4] InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning. In Arxiv 2023.\n\n[5] Uni-Mol: A Universal 3D Molecular Representation Learning Framework. In ICLR 2023.\n\n[6] Geometry-enhanced molecular representation learning for property prediction. In Nature Machine Intelligence 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is clearly written and easy to follow.\n2. The idea is novel and has great potential for expressing the power of LLMs into modeling molecules.\n3. The authors conduct comprehensive experiments, including both on large language model with 1.3B parameters, and light-weight LM with only 125M parameters. This enhances the contribution of the proposed work.", "reasons_to_reject": "1. It seems that the authors did not reveal the computational cost of training Galactica with MolCA's training pipeline, this hinder the understanding of the price in order to gain improvement claimed in the paper.\n\n2. In their experiments, the authors only report results for MolT5 in 1D modeling and MolCA + Galac in 1D+2D modeling, making direct comparisons difficult.", "questions_to_authors": "1. For training two Galactica models in your framework, what is the training cost (e.g., cost of time / gpus) for completing both pretrain stage 1 and stage 2?\n2. Does MolCA has potential to extend to molecule classification and generation tasks? I notice in Table 5, although adding topological information, the improvement of performance is still limited, and is worse than some pretrained molecule encoder, which is trained only on 2D molecular graph data.\n3. Can you provide results of MolT5 + MolCA and Galac with 1D smiles only?", "paper_topic_and_main_contributions": "This paper claims that although current large language model show impressive capacity in understanding molecule, the nature of LLMs to allow only 1D string as input restricted their abilities in further percept 2D topological structure of molecules. To address such problem, the paper propose a framework called MolCA. The proposed framework adopt Q-Former as cross-modal projector to the LLMs, and provide a pretrain pipeline to conduct cross-modal alignment. Empirical study shows excellent abilities of MolCA in improving LLMs' abilities of understanding molecules.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "3PkJd5Xeft", "reviewer": "Reviewer_Cn2f", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review and positive feedback on our submission. We are delighted to learn that you found our idea inspiring and our writing smooth and clear. To further enhance the quality of our submission, we will revise our draft based on each reviewer's comments and suggestions. Thanks again for your support!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "In this paper, authors propose MolCA to enables the LM to understand 2D graphs. This has some inspiration on how to better utilize other modal to improve the performance. Besides, the structure of this paper is coherent and compact, with the author's writing logic flowing smoothly. The graphs are clear, and the experimental results are abundant. Overall, it is well-received.", "reasons_to_reject": "no", "questions_to_authors": "", "paper_topic_and_main_contributions": "Language Models have demonstrated impressive molecule understanding ability on various 1D text-related tasks. However, they inherently lack 2D graph perception.  In this work, the authors propose MolCA to bridge this gap. In details, they represent a graph encoder module to translate the representations of 2D graphs into 1D soft prompts in the text space, and a cross-modal projector to harness the power of existing large LMs for molecule-to-text generation. Besides, they integrate the LM with a uni-modal adapter LoRA to enable its efficient adaptation. The experimental results validate the effectiveness.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper addresses the problem of making LLMs understand 2d molecular structure They propose MoICA which contains (i) a cross-modal projector to connect a graph encoder's representation space and an LM's text space and (ii) a uni-modal adapter to help the LLM to adapt to downstream tasks. They show that MolCA significantly outperforms existing methods on tasks of molecule captioning, IUPAC name prediction, and molecule-text retrieval.

I found this to be a very interesting paper (mainly because it deals with very unique modalities). The reviewers are very positive about this work and have not raised any serious concerns. They agree that the approach is novel and the evaluation is comprehensive. There was a concern about the comparison with related word was not directly comparable but this has been addressed in the response.

Overall, the paper is sound and exciting. I request the authors to include the additional experiments and clarifications provided during the rebuttal into the final version of the paper.
Decision: Accept-Main