Multimodal Molecular Pretraining via Modality Blending
Authors: Qiying Yu
Yudi Zhang
Yuyan Ni
Shikun Feng
Yanyan Lan
Hao Zhou
Jingjing Liu
Venue: ICLR 2024 poster (2024)
Keywords: multimodal molecular pretraining
molecular representation learning
modality blending
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
TL;DR: MoleBLEND, a multimodal self-supervised learning method for molecular that organically integrates 2D and 3D modalities in a fine-grained manner.

Abstract: Self-supervised learning has recently gained growing interest in molecular modeling for scientific tasks such as AI-assisted drug discovery. Current studies consider leveraging both 2D and 3D molecular structures for representation learning. However, relying on straightforward alignment strategies that treat each modality separately, these methods fail to exploit the intrinsic correlation between 2D and 3D representations that reflect the underlying structural characteristics of molecules, and only perform coarse-grained molecule-level alignment. To derive fine-grained alignment and promote structural molecule understanding, we introduce an atomic-relation level "blend-then-predict" self-supervised learning approach, MoleBLEND, which first blends atom relations represented by different modalities into one unified relation matrix for joint encoding, then recovers modality-specific information for 2D and 3D structures individually. By treating atom relationships as anchors, MoleBLEND organically aligns and integrates visually dissimilar 2D and 3D modalities of the same molecule at fine-grained atomic level, painting a more comprehensive depiction of each molecule. Extensive experiments show that MoleBLEND achieves state-of-the-art performance across major 2D/3D molecular benchmarks. We further provide theoretical insights from the perspective of mutual-information maximization, demonstrating that our method unifies contrastive, generative (cross-modality prediction) and mask-then-predict (single-modality prediction) objectives into one single cohesive framework.

Published as a conference paper at ICLR 2024
MULTIMODAL MOLECULAR PRETRAINING MODALITY BLENDING
VIA
Qiying Yu'; Yudi Zhang Yuyan Ni², Shikun Feng', Yanyan Lan Hao Zhou Zh.u Jingjing Liu'
1 Institute for AI Industry Research, Tsinghua University Harbin Institute of Technology
3 Academy of Mathematics and Systems Science, Chinese Academy of Sciences
4 Beijing Academy of Artificial Intelligence
5 Shanghai Artificial Intelligence Laboratory yuqy22 @mai tsinghua. edu. en, zhouhao@aiz tttaeeeeaaii.stsiuuaaaaa edu.
en,
ABSTRACT
Self-supervised learning has recently gained growing interest in molecular mod- eling for scientific tasks such Al-assisted drug discovery. Current studies consider leveraging both 2D and 3D molecular structures for representation learn- th ing. However, relying on straightforward alignment strategies that treat each modality separately, these methods fail to exploit the intrinsic correlation between 2D and 3D representations that reflect the underlying structural characteristics molecules, and only perform coarse-grained molecule-lewe alignment. To de- rive tine-griined alignment and promote structural molecule understanding, introduce an atomic-relation level gllen-theen-pedice self-supervised learning approach, MOLEBIEEND, which first blends atom relations represented by differ- ent modalities into one unified relation matrix for joint encoding, then recovers modality-speciff information for 2D and 3D structures individually. By treating atom relationships as anchors, MOLEB LEND organically aligns and integrates visu- ally dissimilar 2D and 3D modalities of the same molecule fine ane-grained atomic level, painting more comprehensive depiction of each molecule. Extensive exper- iments show that MOLEBLEND achieves state-of-the-ae performance across major 2D/3D molecular benchmarks. We further provide theoretical insights from the maximization, demonstrating that our method
perspective of unifies contrastive, generative (cross-modality prediction) and mausk-then-predee (single-modality prediction) objectives into one single cohesive framework.
INTRODUCTION
Self-supervised learning has been successfully applied to molecular representation learning Xia let al. [2023 Chithrananda et al. 2020), where meaningful representations are extracted from large
amount of unlabeled molecules. The learned representation can then finetuned to support diverse downstream molecular tasks. Early works design learning objectives based on single modality (2D
topological graphs (Hu et al.) [ 2020 Rong et 2020 You 2020), or 3D spatial structures (Zaidi
al. 2022 Liu et al. 2022a Zhou et al.) 2023)). Recently, multimodal molecular pretraining that
exploits both 2D and 3D modalities a single framework (Liu et al. 20226 [Stärk et al. 2022 Liu
al. 2023 Luo al. 2022 Zhu al. 2022) has emerged as alternative solution.
Multimodal pretraining aims to align representations from different modalities. Most existing methods naturally adopt two models (Figure (a)) encode 2D and 3D information separately (Liu et al.)
2022b Stärk et al. 2022 Liu et al. 2023). Contrastive learning is typically employed to attract representations of 2D graphs with their corresponding 3D conformations of the same molecule, and repulse those from different molecules. Another school of study is generative methods that bridge 2D and 3D modalities via mutual prediction (Figure a(ab)), such taking 2D graphs as input predict 3D information, and vice versa (Liu et al. 2022b /Zhu et al. 2022
Liu et al. 2023).
"Equal contribution. Corresponding Author

---

Published as a conference paper at ICLR 2024
(a) Separate Model, Separate
(b) Unified Model, Separate Input
(c) Unified Model, Blended Input (Ours)
Molecule-leve alignment
OR
Ispui Input
2D Encoder
3D Encoder
Unified Encoder
Unifled Encoder
t
OR
Relation-level alignment
Shortest Path
Type
20 Graph
Modality-Blending
vs H
Atom Relations
66
3D Structure
Random Sampling
Figure 1: Comparison on the process of input data. (a) (Liu et al.) 2021a Stärk et al., 2022 Liu
al., 2023) and (b) (Zhu al. 2022) treat different modalities separately, while (c) (ours) blends modalities as input and output. Same atoms (vl, v6) are shared across modalities, while the depictions of atom relationships (shortest path, edge type, 3D distance) are represented by different matrices, which are blended into an integral input for unified pretraining with explicit alignment.
However, these approaches only align different modalities on a coarse-gra coarse-grained molecule-level. The contrastive learning used in most existing methods has been proved lack detailed structural under- standing of the data (Yuksekgonul et al. 2022; Xie et al., 2022), thus missing deep comprehension of the constituting atoms and relations, which plays a vital role representing molecules (Schütt) et al. 2017 Liu et al. (0211b. Besides, all methods consider different modalities as independent signals each model and treat them as separate integral inputs (Figure ba- This practice divides different modalities apart and ignores the underlying correlation between 2D and 3D modalities, only realizing rudimentary molecule-level alignment.
To derive a more fine-grained alignment and promote structural molecular understanding, deeper look into the atom-relation-lewe sub-structures asked for. We observe that although appearing visually distinct and residing in different high-dimensional spaces, 2D molecular graphs and 3D spatial structures are intrinsically equivalent as they are essentially different manifestations of the same atoms and their relationships. The differentiating factor of relationship appears chemical bond or shortest path distance in 2D graph, or 3D euclidean distance in 3D structure. Thus, pivoting around atom relationship and explicitly leveraging the alignment between modalities mutually enhance both 2D and 3D representations can be more natural and effective alignment strategy.
In this work, we introduce a relation-level multimodal pretraining method, MOLEBLEND, which explicitly leverages the alignment of atom relations between 2D and 3D structures and blends input signals from different modalities as one unified data structure pre-train one single model (Fig- ure [c)). Specifically, MOLEBLEND consists of two-stage blend-then then--rediccc training procedure: modality-blente blended encoding and modality -trrgeted prediction. During encoding, we blend different depictions of atom relations from 2D and 3D views into one relation matrix. During prediction, the model recovers missing 2D and information as supervision signals. With such a relation- level blending approach, multimodal molecular information is mingled within unified model, and fine- mine-grained atom-relation alignment in the multimodal input space leads to deeper structural under- standing of molecular makeup. Extensive experiments demonstrate that MOLEBLEND outperforms existing molecular modeling methods across a broad range 2D and 3D benchmarks. We further provide theoretical insights from the perspective of nntuualinformatiion maximization validate the proposed pretraining objective.
Our contributions summarized follows:
2

---

Published as a conference paper at ICLR 2024
We propose align molecule 2D and 3D modalities at atomic-relation level, and introduce MOLEBLEND, multimodal molecular pretraining method that explicitly utilizes the intrinsic correlations between 2D and 3D representations in pretraining.
Empirically, extensive evaluation demonstrates that MOLEBLEND achieves state-of thrtt perfor- mance over diverse 2D and 3D tasks, verifying the effectiveness of relation-level alignment.
Theoretically, we provide decomposition analysis of our objective as explanatory tool, for better understanding of the proposed blend-then-po predict learning objective.
RELATED WORK
Multimodal molecular pretraining (Liu et 2022b; Stärk al., 2022 Zhu al., 2022 Luo et
2022 Liu et al. 2023) leverages both 2D and 3D information to learn molecular representations. It
bears a trade-off between cost and performance, as 3D information is vital for molecular property prediction but 3D models tend to be resource- inourcceintensive during deployment. Most existing methods
utilize two separate models to encode 2D and 3D information (Liu et al. 2022b Stärk et al., [0022]
Liu et al. 2023). Their pretraining methods mostly use contrastive learning (He et al. 22220) which treats 2D graphs with their corresponding 3D conformations as positive views and information from different molecules negative views for contrasting. Another pretraining method uses generative models to predict one modality based on the input of another modality Liu et al. (2022b 2023). Zhu al. (2222) proposes to encode both 2D and 3D inputs within single GNN model, but different modalities are still treated as separate inputs. We instead propose to leverage atom relations as the anchor blend different modalities together as an integral input to a single model.
Masked auto-encoding (Vincent et al. (2008) is widely applied representation learning method (De vlin al. 2019 He et al. 2022) that removes a portion of the data and learns to predict the missing content (mask-therd (hen---tee---reicr)) Multimodal masking approaches in other multimodal learning areas (e.g.. BEiT-3 (Wang al., 2022a), UNITER (Chen et al.) (020b)) directly concatenate different modalities into a sequence, then predict the masked tokens, without explicit alignment of modalities in the input space. Different from them, MOL.BBLEND blends together the elements of different modalities in the input space with explicit alignment.
3 MULTIMODAL MOLECULAR PRETRAINING VIA BLENDING
Molecules are typically represented by either 2D molecular graph or 3D spatial structure. Despite their distinct appearances, they depict a common underlying structure, i.e., atoms and their relationships (e.g., shortest path distance and edge type in 2D molecular graph, and Euclidean distance in 3D structure). Naturally, these representations should be unified organically, instead of treated separately with different models, in order to learn the representation of complex chemical relations underneath. We perform explicit relation-level alignment via blending for unifying modalities.
3.1 PROBLEM FORMULATION
A molecule M can be represented as set of atoms V E 1 along with their relationships R E R2xxxx, where n is the number of atoms, v and r are dimensions of atom and relation feature, the context. In the commonly used 2D graph
respectively. The nature of R can vary depending representation molecules, R is represented by the chemical bonds ε, which are the edges of the 2D molecular graph. In 3D scenarios, R is defined as the relative Euclidean distance D between atoms.
To leverage both 2D and 3D representations, we adopt the shortest path distance Rapd and the edge type encoding Rodge of molecular graph, well as Euclidean distance Rdisstance in 3D space, as three different appearances of atom relations across 2D/3D modalities. And instead of treating each modality separately with individual models, we blend the three representations into a single matrix R2DR3D by randomly sampling each representation for each vector, following a pre-defined multinomial distribution S. Our pre- pre-training objective is to maximize the following likelihood:
max Eg P(Rs Radgey Relistance RIDAAD.S V)
We employ the Transformer model /Vaswani et al. 2017) to parameterize our objective, capitalizing on its ability to incorporate flexible atom relations in fine-grand gininee fashion through attention bias (Raffell
3

---

Published as a conference paper at ICLR 2024
Moodality-Targeter Prediction
Recovery
Flatten Linear, [m, m) i
Fage Type
22
2D Graph
Modality- Blenndd Encoding
v3 v2
H
Outer Product
Atom Rolations
Atom Features,
3D Structure
Blending
GELU & LN
Uneer
30 Dutarne
Unear
Transformer
Input Atoms
Self-
Feed-
Forwer
OO.
Attn
Forwar
Figure 2: Illustration of unified molecular representation learning process, consisting of two steps: 1) modality-blendet encoding, which blends diverse atom relations together and injects it into the self-attention module of Transformer for unified cross-modality encoding; 2) modality-targeted prediction, where atom features encoded by Transformer are transformed into atom relations through an outer product projection module, to recover the diverse relation depictions.
Ke al. 2021 Ying al. 2021). This choice is further supported by recent research demonstrating that single Transformer model can effectively process both 2D and 3D data (Luo al. 2022).
al.
2020 Shaw al. 2018
Transformer Block The Transformer architecture is composed of stack of identical blocks, each containing a multi head self-attention layer and position-wise feed-forward network. Residual connection (He et al., 2016) and layer normalization (Ba et al.) (016)) are applied to each layer. Denote X x x.i........ xx,] as the input to the /-th block with the sequence length n, and each vector 2; E Rd is the contextual representation of the atom position i. d is the dimension of the hidden representations. A Transformer block first computes the multi-head self-attention to effectively aggregate the input sequence X
Multi-Head(X) Connca(hhead head,, W
where head; Attention @ xw XW and h is number of attention heads. W WK wY E Rdxds wo Rdxd are learnable parameter matrices. The attention computation defined as:
QK Vd
Attention K, V) softmax
V
(3)
Generally, given input X the 1-th block works follows:
X LayerNorm X + MMulti-Hea((XX)) (X)))
(4)
H1 LayerNorm X + X
(5)
where W E Rdxds W! xd and d is the intermediate size of the feed-forward layer.
3.2 LEARNING OBJECTIVE
To facilitate fine- -re-grained alignment and organic integration of different depictions of atoms and their relations across 2D/3D spaces, design new blend-then deen-ireeict' training paradigm that consists of two steps: 1) modality- -lended encoding that encodes a molecule with blended information from different modalities; and 2) modality-targette prediction that recovers the original 2D and 3D input.
4

---

Published as a conference paper at ICLR 2024
The pre-training process is illustrated in Figure The core idea is to bind different modalities together at a granular level by blending relations from multiple modalities into an integral input from the get-go, to encourage the model to discover fundamental and unified relation representations across heterogeneous forms.
Modality-blendee Encoding Multimodal learning aims learn the most essential representations of data that possess inherent connections while appearing distinctive between different modalities. In the context of molecules, atom relationships are the common attributes underpinning different representations across 2D/3D modalities. This motivates us to leverage relations as anchors, to align both modalities a fine-grained manner that blends multimodalities from the very beginning.
We adopt three appearances of relations across 2D and 3D modalities following (Luo et al. 2022): shortest path distance, edge type, and 3D Euclidean distance. For each atom pair (i,j). 1/ represents the shortest path distance between atom é and j. We encode the edge features along the shortest path between and j as the edge encoding, V
+ W, en where en E Ld are features of edges on the shortest path between and j. wn E Rd are (e1, e....., eN),
learnable parameters. Following (Zhou et al. 2023 Luo al. (202). we encode Euclidean distances of an atom pair (i,j) with Gaussian Basis Kernel function (Schölkopf et al.
1997
ch = G(A(d) 88);; (4k ok), k = 1,...... K
(6)
viti
GELU(Ç" W W 5 15-=11 -[I
(7)
Distance
is the affine transformation with learnable parameters and 3, and 2 (d n)²) (d-+)²² is the Gaussian density function with parameters μ and K is the number of Gaussian Basis kernels. W.D RKK W3 E R are learnable parameters. SPD, Vrage, Distance denote the three relation matrices of all atom pairs, with the same shape x
where A(d; 7, 7
7
(d:: (4, 0) V2na exp
Different from existing works that separately feed one of these relations into different models, we blend them together from the get-go and randomly mix them into one relation matrix, which is then fed into one single model for molccule encoding. Specifically, we first define a multinomial distribution S with probability vector p (p1, P2, Pa). For each position (i,j) in the matrix, draw sample S E (1,2,3) 2) following the probability distribution p, then determine the corresponding element of the blended matrix follows:
s k
v 2D&3D
V SPD 11 +
Edge 11 + Distunce 11, where 1 =
(8)
otherwise
where each position (i,j) randomly selects its element from one of the w W W the process finishes, distinct relation manifestations (Wspp, V V Distance) across modalities are blended into a single modality- blended matrix 2DDADD E R'xx without overlapping sub-structures, to represent the inter-atomic relations.
Edge Distance After
We inject this modality- blended relation 2DD&DD into the self-attention module, which captures pair-wise relations between inputs atoms, provide complementary pair-wise information. This practice also similar to the relative positional encoding for Transformer (Raffel et al. 2020):
QKT
Attention Q, K, V) = softmax
V2D&DD
Vd
With modality-blendiin we explicitly bind different modalities together at fine tme-ggriied relation level, which will help the model better integrate and align modalities at fine- tinengrained level.
-trgeed Prediction The model recovers the full Rapd, Redge and Rdistance its training objectives. The intuition is, if the model can predict different types of atom relations, like shortest path the molecular graph 3D Euclidean distance, given single mixed representation, this cross-modality representation must have captured some underlying integral molecular structure.
Specifically, after modality- blended encoding, obtain contextual atom representations X++1 Rd encoded by an L-layer Transformer. We propose an outer product projection module to transform the atom representations into atom relations. The representations X XL+1 are first
5

---

Published as a conference paper at ICLR 2024
linearly projected to smaller dimension 32 with two independent Linear layers W., W, Rmxdd The outer products are computed upon the transformed representations, which are then flattened and projected into the target space with modality-targeete head W Whead E Rex== The relation computation between the i-th and j-th atoms formulated as follows:
Oij G(WiX @ G( E Rmmm
(10)
Zij W head Flatten(o:,) R
(11)
where G(.) = (oyerrorm((GELU()) We now obtain the modality- relation matrix Z E |Xxxxx where depends on the targeted task. The predictions of shortest path distance and edge type are formulated as classification tasks, where c is the number of possible shortest path distance or edge types. For predicting 3D distance, we formulate it as a 3-dimensional regression task, and the regression targets are the relative Euclidean distances in 3D space.
Noisy Node as Regularization Noisy node (Godwin et al., 2022 Zaidi et al. 2022 Luo al. 2022) incorporates auxiliary loss for coordinate denoising in addition to the original objective, which has been found effective improving representation learning. We also adopt this practice as an additional regularization term, by adding Gaussian noise the input coordinates and requiring the model to predict the added noise.
3.3 FINETUNING
The trained model can be finetuned to accept both 2D and 3D inputs for downstream tasks. For scenarios where large amount of 2D molecular graphs is available while 3D conformations are too expensive to obtain, the model can take only 2D input finetune the model. Formally, given shortest path distance Rypa, edge type Redge and atom types V as available 2D information, we define yan as the task target, K as the number of training samples, and ((:,)) .) as the loss function of the specific training task. The 2D finetuning objective is then defined as:
K
C2D K
Σ e (I(R Redges V²), yin) k=1
(12)
When it comes to scenarios where 3D information is obtained, we propose to incorporate both 2D and 3D information model input, as generating 2D molecular graphs from 3D conformations is free and can bring in useful information from 2D perspective. The multimodal input injected into the self-attention module that captures wair-wise relations:
QKT
Attention Saantion(Q... K. V) softmax
+ SPD + V Vadge + L Distance
V
(13)
Vd
K Σ km
CaD
e (f(R)p) Ridger Russtanee V*) y
(14)
K
This practice is unique in utilizing information from multiple modalities for a single-modality task, which is infeasible in previous 3D (Zaidi et al.) 2022) multimodal methods with separate models for different modalities (Liu et al. 2022b; Stärk et al. 2022] Liu et al., (023). Empirically, we find that the integration of 2D information helps improve performance. we hypothesize that: 1) 2D information, such as chemical bond on a molecular graph, encodes domain experts' prior knowledge and provides references to 3D structure; 2) 3D structures obtained from computational simulations can suffer from inevitable approximation errors (Luo et al.) 2022) which are avoided in our approach.
3.4 THEORETICAL INSIGHTS
In this section, we present a theoretical perspective from mutual information (MI) maximization for a better understanding of the sandd-then-predicee process. We demonstrate that this approach unifies existing contrastive, generative (inter- modality prediction), and mask-then- tthen-predictt (intra-modality prediction) objectives within single objective formulation.
For simplicity, we consider two relations, denoted as Rep = (azj)axn and RaD = (bzz) munn- Their elements are randomly partitioned into two parts, represented RaD [A1, A2), RaD [B1, B2),

---

Published as a conference paper at ICLR 2024
such that A, shares identical elements indexes with B,, i {1,2]. The blended matrix is denoted as R2DE3D = [A1, B2]-
Proposition 3.1 (Mutual information Maximization) The training process with modamimy- maximizes the lower bound of the following mutual information: EşI A1. [B1;; A1, B2) The proof can be sound in Appendix [.2.4
Proposition 3.2 (Mutual Information Decomposition) The mutual information I [[2;; A1, B2) [B;; A1, B2) can be decomposed into two components below. The first one corresponds to the objectives of contrastive and generative approaches. The second component, the primary focus of our research, represents the musk-then-predic objective (proof in
i Proposition B. 1 in Appendix):
[[A2; A1, B2) - [B1; A1, B2)
[AA:; + 1(A2; Ba)
I(A1; B1 B2) + (A2; Ba(AA)]
contrastive and generative
conditional conmassive and goneranive
(15)
A2) + [(B;; B2)
I(A1; B2) [B;; B2(Ai)
mask-lteen-prdierr
saaitiordal chen phrndii prodic
The first part of Equation 15 corresponds to existing (conditional) contrastive and generative methods, which aim to maximize the MI between two corresponding parts (A, with B,, i E {1,2}) across two modalities (see Appendix [ and B.2.3 for the detailed proof). The second part represents the (multimodal) mask-then-predie objectives, focusing on maximizing the mutual information between the masked and the remaining parts within single modality (refer to Appendix B.2.2 for details).
This decomposition illustrates that our objective unifies contrastive, generative (inter-modality pre- diction), and mask-then-predict (intra-modality prediction) approaches within single cohesive alend-then-ao theen-tredcc framework, from the perspective of MI maximization. Moreover, this approach fosters enhanced gross-modal interaction with an innovative multimodal mask-then- then-predict target.
EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Datasets. For pretraining, we use PCQM4M22 dataset from the OGB Large-Scale Challenge (Hu
al. 202T), which includes 3.37 million molecules with both 2D graphs and 3D geometric structures.
To evaluate the versatility of MOLEBLEND, we carry out extensive experiments on 24 molecular tasks
with different data formats across three representative benchmarks: MoleculeNet (Wu et al. (017))
(2D, 11 tasks), QM9 quantum properties (Ramakrishnan et al. 2014) (3D, 12 tasks), and PCQM4Mv2
humo-lumo gap (2D). Further details about these datasets can be found in the Appendix C.1
Baselines. We choose the most representative 2D and 3D pretraining baselines: AttrMask (uu et al. 2020), ContextPred (Hu et al. (020), InfoGraph (Sun et al.] 2020). MolCLR (Wang et al. 2022b). GraphCL (You et al. (020), GraphLoG (Xu et al. 2021), MGSSL ZZhng et al. (202 1). as well as recently published method Mole-BERS (Xia et al. 2023) and GraphMAE (Hou et al. (022) as 2D baselines. In addition, we adopt GraphMVP (Liu et al. 2022b), 3D InfoMax (Stärk et al. (022). UnifiedMol Zhu et al. (022) and MoleculeSDE (Liu et al. (223) as multimodal baselines. As most baselines adopt GNN backbone,
further implement two close-related multimodal pretraining baselines, 3D Infomax and GraphMVP, under the same Transformer backbone as we use, to fairly compare the effectiveness of pretraining objective.
Backbone Model. Following Ying et al. 202 I Luo etal 2022), we employ 12-layer Transformer of hidden size 768, with 32 attention heads. For pretraining. we use AdamW optimizer and set (B,, B2) to (0.9, 0.999) and peak learning rate to le-5. Batch size is 4096. We pretrain the model for I million steps with initial 100k steps as warm-up, after which learning rate decreases to zero with cosine scheduler. The blending ratio p is 2:2:6, and the ablations on p can be found in Appedix A.3
4.2 EVALUATION ON 2D CAPABILITY
We evaluate MOLEBLEND on MoleculeNet, one of the most widely used benchmarks for 2D molecular property prediction, which covers molecular properties ranging from quantum mechanics
7

---

Published as a conference paper at ICLR 2024
Table I: Results molecular property classification tasks (with 2D topology only). We report ROC- AUC score (higher is better) under scaffold splitting, Transformer impl. represents implementation under the same Transformer backbone as MOLEBLEND. Results in gray are evaluated under a different protocol.
Pre-training
Backbone Type
Methods
BRRP
Tox21
ToxCast
SIDER
ClinTox
MUV1
HTV
Bace
AttrMask
(Hu et al.
2020
GNN
65.0=2.3
74,8++0.
62.940.1
61.2+0.1
87.++11
73.4+20
76.8+0.5 79.7-0.3 72.68
ContexiPred
2020
65.7=0.6
74.240.0
62.510.3
62.2::.5
77.220.8
75.321.5
77.12088 76.0=20 71.28
GraphCT
You ct
2020
GNN
69.7=0.6 73.9+40.6
62,440.5
60.5=008
76.0=2.6
69.8+2.66
78.5+1.2
75.4-14 70.78
InfoCrraph
Sun
2020
ONN
67.5=0.1 73.240.4
63,710.5
59.92033
76.5210
74.120.7
75.1209
77,8=0.8 70.98
GROVER
Rong et
2020
Transformer
70.0+0010 74.340.1
65.440.4
6488=066
81.2+3.0
67.3+1.8
62.5+09
82.6-077 71.01
MolCT.R
Wang cl
1022h
GNN
66.6-188 73.0+0.1
62,940.3
57,5+1.7
86.1+0.9
72.5+2.3
76.2+155
71,5-311 70.79
GraphLoGi
Xt al.
2021
72.520.8 75.720.5
63.520.7
61.2=111
76.7=3.
76.211
77.8208
83.5-12 73.40
MGSSL
7 Thaggete et
2021
GNN
69.7-0.9 76.540.3
64.140.77
61.8+088
80.7+2.11
78.7+1.5
78.8+1.2
79.1-09 73.70
GrapiMAE
Hou et al.
2022
GNN
72.0=0.6 75.540.6
64.120.3
60.3=111
823=112
76.3#2.4
77.2=100 83.1=09 73.85
Mole-RFRT
et
71.9-1.6
76.8+0.5
64.340.2
6288-111
78.9+3.0
78.6+1.8
78.2+0.8 80.8 74.04
3D InfoMax
Stärk
2022
GNN
69.1=10 74:520.7
64.42088
60.620.7
79.9=34
74.4224
76.1=1.3
79.7=1.5 72.34
GrapMMPP eiu al.
2022b
GNN
68.5.0.2 74.5.10.4
62.710.11
62.31.66
79.012.5
75.011.4
74.8114
76.8_.11 71.69
MoleculeSDB Liu et
2023
GNN
71810.7
76.8.0..3
65.0.0.2
60.8...3
87.0.0.5
80,910.3
78.81099
79.5_2.1 75.07
Transformer from scratch
Transformer 69.4=111
74.240.3
62.640.3
65.8=0.3
90.3=009
71.3=0.8
76.2=0.6
79.5=0.2 73.66
3D InfoMax (Trnsformer impl.)
Transformer 70.411.0
75,5.10.5
63.110.7
64.110.1
89.1122
72.81.0
74.90..3
80.7..06 73.91
GraphMVP (Transformer impl.)
Transformer
71.5=13
76.1+0.9
64.340.6
64.7=0.7
89.9=099
74.9112
76.0=0.6
81.5=1.2 74.86
MOLEBLEND
Transfrmer
73.0-0.8 77.8.10.8 66.1.1..0 64.10.3 87.610.77 77.211..3 79.0.0.8 83.7 1.4 76.16
and physical chemistry biophysics and physiology. We use the scaffold split (Wu et al. report the mean and standard deviation of results of 3 random seeds.
2017 and
Table D presents the ROC-AUC scores for all compared methods on eight classification tasks. Remarkably, MOLEBLEND achieves offte-of-the-ar performance in 5 out of 8 tasks, with significant margins in some cases (e.g., 83.7 v.s. 81.5 on Bace). Note that all other multimodal methods (3D Infomax (Stärk et al. 2022), GraphMVP (Liu et al. 2022b), MoleculeSDE (Liu et al. (203)) utilize two separate modality-speciinn models, with contrastive learning as one of their objectives. In contrast, MOLEBLEND models molecules in unified manner, and perform 2D and 3D alignment in fine- grained relation-level, demonstrating superior performance. MOLE BLEND also outperforms all 2D baselines (upper section of the table), demonstrating that incorporating 3D information helps improve the prediction of molecular properties. summarize the performance of different methods on three regression tasks of MoleculeNet, which substantiates the superiority of MOLEBLEND.
4.3 EVALUATION ON 3D CAPABILITY
We use QM9 (Ramakrishnan et al. 2014) dataset to evaluate the effectiveness of MOLEBLEND on 3D tasks. QM9 quantum chemistry benchmark with 134K small organic molecules. contains 12 tasks, covering the energetic, electronic and thermodynamic properties of molecules. Following (Thölke & Fabritiis| 2022). we randomly split 10,000 and 10,831 molecules as validation and test set, and use the remaining molecules for finetuning. Results are presented in Table 2 evaluated on MAE metric (lower is better). MOLEBLEND achieves state-of-the-art performance
Table Results on QM9 datasets. Mean Absolute Error (MAE, lower is better) is reported.
Pro-training Methods
Alpha
Gap 1 HOMO 1 LUMO L Mu
Cv
G298 | H298 J R2
U298 L U0 4 Zpvo
Distance Prediction (Luu al.
2022a
0.065
45.87
27.61
23.34
0.031
0.033
14.83
15.81
0,248
15.07
15.01
1.837
3D InfoGGrahh
LLu
2027a
0.062
45.96
29.29
24.60
0.028
0.030
13.93
13.97
0.133
13.55
13.47
1.644
3D InfoMax
Stark et al.
2022
0.057
42.09
25.90
21.60
0.028
0.030
13:73
13.62
0.141
13.81
13.30
1.670
GraphMVP Liu et
2022h
0.056
41.99
25.75
21.58
0.029
13.43
13.31
0.136
13.03
13.07
1.609
MoleculeSDE
Liu et
2023
0.054
41.77
25.74
21.41
0.026
0.028
13.07
12.05
0.151
12.54
12.04
1.587
MOLEBLEEN
0.060
34.75
21.47
19.23
0.037
0.031
12.44
11.97
0.417
12.02
11.82
1,580
8

---

Published as a conference paper at ICLR 2024
Table 3: Ablation studies on pretraining objectives. The best and second best results marked by bold and underlined.
traiinn Methods
BBBP 1 Tox21 1 ToxCast
SIDER
Clin'Tox f MUV
HIV f
Bace f U298 4
U0 4
Noisy-Node
68.50
76.25
65.48
63.71
83.28
78.80
79.13
82.72
14.31
13.80
Blend- Serid-the--Preee Predict
71.59
75.61
65.93
64.58
90.82
76.81
79.74
83.53
14.56
15.35
MOLEBLEND
73.00
77.82
66.14
64,90
87.62
77.23
79.01
83.66
12.02
11.82
among multimodal methods on out of 12 tasks, some which with large margin (e.g., Gap, HOMO, LUMO), demonstrating the strong capability of our model for 3D tasks.
4.4 ABLATION STUDIES
Pretraining Objectives Table 3 studies the effect of different pretraining objectives: noisy-node, tlend-then-predict, and weend-then-prediee with noisy-node as regularization (MOLEBLEND), We observe that in most tasks, combining blend- then-predict and noisy-node yields better repre- sentations. In 2D scenarios, we find that blend- then-predict outperforms noisy-node on out of 8 tasks studied, demonstrating its strong ability to process 2D inputs. While on 3D tasks (U298 and U0), blend-then-predict typ- ically performs worse than noisy-node. This is because noisy-node is pure 3D denoising task, which makes it more suitable for 3D tasks.
Table 4: Ablation studies on blending vs masking.
Method
BBBP
BACE
Tox21 f
ToxCast f
SPD mask
75.59
62.82
Edge mask
69.02
81.97
76.01
63.81
3D mask Blending
67.60
75.65
63.28
71.68
83.41
76.58
65.46
Blending Single-modality Mask-then-Predicc Table the effect of multimodal blending compared to single-modality mask-then-predice (SPD, Edge, and 3D mask). We trained all models for 200K steps, keeping all settings consistent except for the learning objective. The results demonstrate that modality blending achieves better performance over modality-spccifie mask-then then-predict.
Finetuning Settings When 3D molecular informa- tion is provided, we propose to incorporate both 2D topological and 3D structural information into the model, as generating 2D molecular graphs from 3D conformations computationally inexpensive. Table 5 demonstrates that the inclusion of 2D information leads hy-
Table 5: Ablation studies on fintuning set- tings of 3D tasks.
Finetune Settings
Alpha
HOMO J
3D
0.066
23.62
0.042
to noticeable improvement in performance. pothesize that this is due to the fact that 2D information encodes chemical bond and connectivity on a molecular graph, which is grounded in prior knowledge of domain experts and contains valuable references to 3D structure. Note that this practice is a unique advantage of MOLEBIEEND, as we pretrain with both 2D and 3D information blended previous multimodal methods that utilize two distinct models for 2D and 3D modalities.
3D + 2D
0.060
21.47
0.037
single input into unified model, which is not feasible in
CONCLUSION
We propose MOLEBLEND, novel relation-level self-supervised learning method for unified molecu- lar modeling that organically integrates 2D and 3D modalities in a fine- fine-grained manner. By treating atom relations the anchor, we blend different modalities into integral input for pretraining, which overcomes the limitations existing approaches that distinguish 2D and 3D modalities as independent signals. Extensive experimental results reveal that MOLEBLEND achieves state-of-the-an performance on a wide range of 2D and 3D benchmarks, demonstrating the superiority of fine -ne-gedined alignment of different modalities.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
This work is supported by the National Key R&D Program of China (2022ZD0160501), Natural Science Foundation of China (62376133) and Beijing Academy of Artificial Intelligence (BAAI).
REFERENCES
Alexander A Alemi, lan Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. arXiv preprint arXiv:1612.00410,, 2016.
Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saun- shi. A theoretical analysis of contrastive unsupervised representation learning. arXiv preprint arXiv: vrXiv:/902.099 /902.09229, 2019.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey Hinton. Layer normalization. arXiv preprint arXiv:1607.064450, 2016.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020a.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. UNITER: universal image-text representation learning. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.), Computer Vision ECCV 2020 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXX, volume 12375 of Lecture Notes in Computer Science, pp. 104 120. Springer, 2020b. doi: 10. 1001/ 778-3-030-58577-11 [RLpttts:://oi.
10. 10000000000000077700000000---170
Scyone Chithrananda, Gabriel Grand, and Bharath Ramsundar. Chemberta: Large-scale self- supervised pretraining for molecular property prediction. CoRR, abs/2010,09885, 2020. URL
https: org/ abs/ 2010. 09885
Thomas M. Cover and Joy A. Thomas. Elements of Information Theory, pp. 23. Wiley Online Library, 2nd edition, 1991.
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04885, 2018.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume (Long and Short Papers), pp. 4171- 4186. Association for Computational Linguistics, 2019. doi: 10. 1.11853/11/119-1425 URL httpp////aaaloogg11.111515//////
Jonathan Godwin, Michael Schaarschmidt, Alexander L. Gaunt, Alvaro Sanchez-Gonzalez, Yulia Rubanova, Petar Velickovic, James Kirkpatrick, and Peter W. Battaglia. Simple GNN regularisation for 3d molecular property prediction and beyond. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
URL httl:/ttss//ppeeeeviieeee net Eorum www.mmdddwwwwwwwewedmommm
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 9729-9738. 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16000-16009, 2022.
10

---

Published as a conference paper at ICLR 2024
Zhenyu Hou, Xiao Liu, Yukuo Cen, Yuxiao Dong, Hongxia Yang, Chunjie Wang, and Jie Tang. Graphmae: Self-supervised masked graph autoencoders. In Aidong Zhang and Huzefa Rangwala (eds.), KDD '22: The 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Washington, DC, USA, August 14 18, 2022, pp. 604.604 ACM, 2022. doi: 10.1 10.1145/3534678. 3539321. URL Inttps://dnn
0ttp///.111111531131131 3539321
Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Perey Liang, Vijay S. Pande, and Jure Leskovec. Strategies for pre-training graph neural networks. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. https:://ppeneeiiecee
net wor.mmidimmmmmme HJ WH
Weihua Hu, Matthias Fey, Hongyu Ren, Maho Nakata, Yuxiao Dong, and Jure Leskovec. OGB-LSC: A large-scale challenge for machine learning on graphs. In Joaquin Vanschoren and Sii-Kit Yeung (eds.). Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual, 2021. URL
neur ps. cc ptcp/pappr//221/ woweet-roon...... how.issttrrrt--oooo......tt
Guolin Ke, Di He, and Tie- Liu. Rethinking positional encoding in language pre-training. In
9th International Conference Learning Representatiins, ICLR 2021, Virtual Event, Austria,
May 3-7, 2021. OpenReview.net, 2021. URI. htttps://openreviit
net forum?ic
09 52
Lingpeng Kong, Cyprien de Masson d' Autume, Wang Ling, Lei Yu, Zihang Dai, and Dani Yogatama. A mutual information maximization perspective of language representation learning. arXiv preprint arXiv:1910.083500 2019.
Ralph Linsker. An application of the principle of maximum information preservation to linear systems. Advances in neural information processing systems, 1, 1988.
Shengchao Liu, Hanchen Wang, Weiyang Liu, Joan Lasenby, Hongyu Guo, and Jian Tang. Pre- training molecular graph representation with 3d geometry. arXiv preprint arXiv:2110.07728. 2021a.
Shengchao Liu, Hongyu Guo, and Jian Tang. Molecular geometry pretraining with se (3)-invariant denoising distance matching. arXiv preprint vrXiv:2206.13602 2022a.
Shengchao Liu, Hanchen Wang, Weiyang Liu, Joan Lasenby, Hongyu Guo, and Jian Tang. Pre- training molecular graph representation with 3d geometry. In International Conference on Learning Representations, 2022b.
Shengchao Liu, Weitao Du, Zhiming Ma, Hongyu Guo, and Jian Tang. A group symmetric stochastic differential equation model for molecule multi-modal pretraining. In International Conference on Learning Representationss 2023.
Yi Liu, Limei Wang, Meng Liu, Xuan Zhang. Bora Oztekin, and Shuiwang Ji. Spherical message passing for 3d graph networks. CoRR, abs/2102.05013, 2021b. URL
abs/ 22122112..1 05013
Shengjie Luo, Tianlang Chen, Yixian Xu, Shuxin Zheng, Tie- Yan Liu, Liwei Wang, and Di He. One transformer can understand both 2d & 3d molecular data, arXiv preprint arXiv:22/0.01763 2022.
David McAllester and Karl Stratos. Formal limitations on the measurement of mutual information. In International Conference on Artificial Intelligence and Statistics, pp. 875-884. PMLR, 2020.
Yuyan Ni, Yanyan Lan, Ao Liu, and Zhiming Ma. Elastic information bottleneck. Mathematics, 10 (18):3352, 2022.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint /807.03377 2018.
Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In International Conference on Machine Learning, pp. 5171-5180. PMLR, 2019.
11

---

Published as a conference paper at ICLR 2024
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485-555 2020.
Raghunathan Ramakrishnan, Pavlo O Dral, Matthias Rupp, and O Anatole Von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific data, 1(1):--7, 2014.
Yu Rong, Yatao Bian, Tingyang Xu, Weiyang Xie, Ying Wei, Wenbing Huang, and Junzhou Huang. Self-supervised graph transformer large-scale molecular data. Advances in Neural Information Processing Systems, 33: 12225559257771 2020.
Bernhard Schölkopf, Kah Kay Sung, Christopher J. C. Burges, Federico Girosi, Partha Niyogi, Tomaso A. Poggio, and Vladimir Vapnik. Comparing support vector machines with gaussian kernels to radial basis function classifiers. IEEE Trans, Signal Process., 45(11))2758-2765 1997. doi: 1011008.6600000 URL pattps://ddi..o
1109/711118888 650102
Kristof Schütt, Pieter-Jan Kindermans, Huziel Enoc Sauceda Felix, Stefan Chmiela, Alexan- dre Tkatchenko, and Klaus-Rober Müller. Schnet: A continuous-filee convolutional neu- ral network for modeling quantum interactions. In Isabelle Guyon, Ulrike von Luxburg. Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neu- ral Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 991-1001, 2017. URL 303ed4c699 (ww444423333239342 3305 3050
neurips oc/ppaper/2017/hahh Abst - html
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representations. In Marilyn A. Walker, Heng Ji, and Amanda Stent (eds.), Proceedings of the 2018 Conference of the North American Chapter the Association for Computational Linguisties: Human Language Technologies, NAACL-HLT, New Orleans. Louisiana, USA, June 1-6, 2018, Volume 2 (Short Papers), pp. 464-468. Association for Computational Linguistics. 2018. doi: 10.18653/v1/ n18-2074. URL httph: //doi. http:///ooooo///./////o 0r./10. 1865885555 11w53////nnni - 2074
Hannes Stärk, Dominique Beaini, Gabriele Corso, Prudencio Tossou, Christian Dallago, Stephan Günnemann, and Pietro Liò. 3d infomax improves gnns for molecular property prediction. In International Conference on Machine Learning, pp. 20479-20502. PMLR, 2022.
Fan-Yun Sun, Jordan Hoffmann, Vikas Verma, and Jian Tang. Infograph: Unsupervised and semi- supervised graph-level representation learning via mutual information maximization. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL
net forum?ict
EllfF2NYN
Philipp Thölke and Gianni De Fabritiis. Equivariant transformers for neural network based molecular potentials. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL wwtp//pweeeeew.eemeeeeeeee net/
9wrRB
Naftali Tishby, Fernando Pereira, and William Bialek. The information bottleneck method. arXiv preprint physicc/0004405,,220 2000.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pp. 1096-1103, 2008
Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, and Furu Wei. Image as a foreign language: Beit pretraining for all vision and vision-language tasks. CORR, abs/2208. 10442, 2022a. doi: 10.485500/rXxiv22208. 10442. URL https: /ooi 48550/0.
2208. 10442
12

---

Published as a conference paper at ICLR 2024
Yuyang Wang, Jianren Wang, Zhonglin Cao, and Amir Barati Farimani. Molecular contrastive learn- ing of representations via graph neural networks. Nat. Mach. Intell. 4(3):279 287, 2022b. doi: 10. 1038/s42256-022- 022-04477 BRLh:tps: http:////......o.11111110. 10300...0301100 11w.0100100002220000000200003010
Zhenqin Wu, Bharath Ramsundar, Evan N. Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh Pappu, Karl Leswing, and Vijay S. Pande. Moleculenet: A benchmark for molecular machine learning. CoRR, abs/1703.00564, 2017. [tttt:/://axxix org/ abs /1003 00564
Jun Xia, Chengshuai Zhao, Bozhen Hu, Zhangyang Gao, Cheng Tan, Yue Liu, Siyuan Li, and Stan Li. Mole-bert: Rethinking pre-training graph neural networks for molecules. 2023.
Zhenda Xie, Zigang Geng, Jingcheng Hu, Zheng Zhang. Han Hu, and Yue Cao. Revealing the dark secrets masked image modeling. arXiv preprint arXiv:2205. 13543, 2022.
Minghao Xu, Hang Wang, Bingbing Ni, Hongyu Guo, and Jian Tang. Self-supervised graph-level representation learning with local and global structure. In International Conference on Machine Learning, pp. 11548-15558 PMLR, 2021.
Chengxuan Ying, Tianle Cai, Shengjie I Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tie-Yan Liu. Do transformers really perform badly for graph representation? Advances Neural Information Processing Systems, 34:28877-28888 2021.
Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tiee -ien Lin (eds.), Advances in Neural Information Processing Systems 33: Anmual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. /att:tts:: / neurips. wttpspccpppppppeeea .htmm
Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-language models behave like bag-of-words models, and what to do about it? arXiv preprint arXiv:2210.01953 2022.
Sheheryar Zaidi, Michael Schaarschmidt, James Martens, Hyunjik Kim, Yee Whye Teh, Alvaro Sanchez-Gonnzalez, Peter Battaglia, Razvan Pascanu, and Jonathan Godwin. Pre-training via denoising for molecular property prediction. arXiv preprint arXhv:2206.00133 00103, 2022.
Zaixi Zhang, Qi Liu, Hao Wang, Chengqiang Lu, and Chee- Kong Lee. Motif-based graph self- supervised learning for molecular property prediction. Advances in Neural Information Processing Systems, $4:15870-15882 2021.
Gengmo Zhou, Zhifeng Gao, Qiankun Ding, Hang Zheng, Hongteng Xu, Zhewei Wei, Linfeng Zhang, and Guolin Ke. Uni-mol: A universal 3d molecular representation learning framework. 2023.
Jinhua Zhu, Yingce Xia, Lijun Wu, Shufang Xie, Tao Qin, Wengang Zhou, Houqiang Li, and Tie-Yan Liu. Unified 2d and 3d pre-training of molecular representations. arXiv preprint arXiv:2207.08800 2022.
13

---

Published as a conference paper at ICLR 2024
A EXPERIMENTS
A.1 BASELINE RESULTS
The baseline results of Gr Liu et al. (2021a), MoleSDE Liu et al. (2023). GraphCI You
et al. (020), GraphMAE Hou et (2022), GraphLoG (Xu al. 2021). MGSSL (Zhng al.
2021) are from their own paper. Results of AttrMask (Hu et al. 20220) Contex(Pred (Hu et al. 2020
InfoGraph Sun al. (2020), MolCLR Wang al. (202bb) are from MoleculeSDE Liu al. 2023
Results of MoleBERT (Xia et al. 2023). 3D Infomax (Stärk et al. (2022) are from MoleBEKT. The
results of GROVER Rong et al. (2020) are from Uni-Mol Zhou al. (2023).
A.2 MOLNET REGRESSION TASK
Table O presents the performance of different methods on three regression tasks of MoleculeNet. In all these tasks, MOLEBLEND achieves state-of-the-art performance, further substantiating the superiority of unified fine-grained molecular modeling.
Table 6: Results molecular property prediction regression tasks (with 2D topology only). We report RMSE (lower is better) for each task.
Pre-training Methods
ESOL +
FreeSolv |
Lipo 4
AttrMask HHu 2020
1.112+00.08
0.730+000044
ContextPred Huu et al. 2020
1.196:10.037
0.702+00.000
GROVER basse Rong et al. 2020
0.983+00.000
2.176+0.052
0.817++0.008
MoICLR (Wang et al. 2022b)
1.201
2.594-10.249
0.6914:0.004
3D InfoMax Stärk et al. 2022
0.894+0.028
2.337=0.227
0.695=t0.012
GraphMVP
Liu al.
2022b)
1.029+00003
0.681+0.0010
MOLEBLEND
0.821+-0.026
1.910
0.638+00.004
A.3 ABLATIONS ON BLENDING RATIO
Table presents ablations the relation blending ratio, showing that model performance is robust to the random ratio of multinomial distribution. In these experiments, we trained all models for 200K steps, maintaining other settings unchanged (e.g., learning rate consistent), with the exception of the blending ratio.
Furthermore, we have observed that higher 3D distance ratio (referring to the bottom three rows in the table) sometimes performs better than lower ratio (top row 4:4:2 ratio). This suggests that inclusion of 3D information is potentially more important for enhancing the model's understanding of molecular properties. However, it is worth noting that the disparity in performance between these ratios relatively minor.
Table Ablations on the blending ratio.
SPD:Edge:3D (p)
BBBP
BACE f
Tox21
ToxCast
Lipo |
4:4:2
72.25
82.17
76.23
66.70
0.7544
3:3:4
72.34
82.47
77.19
66.16
0.7505
2:2:6
72.52
82.89
76.15
66.58
0.7511
1:1:8
72.45
82,43
76.46
66.57
0.7478
14

---

Published as a conference paper at ICLR 2024
B THEORETICAL ANALYSIS
In the following sections, we follow common notations(Cover & Thomas 1991). using uppercase letters to represent random variables and lowercase letters represent samples of the random variables.
B.1 MISSING PROOFS
Lemma B.1 (Chain rule of mutual information(Cover & Thomas 1991)
T(X1, X2; V) T(XI:V) + (X2:V|X1
(16)
Proof
((x;Y)+((X)))))) +(X;)) = Eplan.v) log p(x1, + Bp(2))221) log p v
p( p(x1)p(()
Ep(x) [log p(11,8) p(12))/01)
p(x1)p(y) p(a22 (p(p)m)
(17)
-Epla m22))) log p( 1)p y x1)
p(y)p(22,00 21 21)p(y,
= Ep(r
llo p(y)p(z2, (1) x1)
1(X1, 1(X1,
Proposition B,1 (Mutual Information Decomposition) The blend-and anddddedd--dde method is maximiz- ing the lower bound of the mutual information target below, which can be further divided into two parts.
T(A2; A1, B2) + I(Bi; A2)
2 [[A1;; B1) + I(A2; B2) + [(A; BB) + I(A2; B2[Ai)]
(18)
2 [(A1; + I(B1; B2) + (Aj; B2) + I(BI; (2]A1)
Proof Firstly, we provide the decomposition of first term in equation 18 i.e. T(A2: B2). By using Lemma B.1 and letting X1 = A,, X2 B2 and Y = A2, we have
I(A;; A1, B2) = I(A1; A2) + [(A;; B1)
(19)
Again use Lemma B.1 and let X1 = B2, X2 = Ai and Y = Ag, then we have
I(A2; A1, B2) = [(2; A2) + T(A A1 B2).
(20)
From equation 19 and equation 20 have
I(A;; A1, B2)
2 [T(A; A2) + I(A;; B2(A1) I(B;; A2) + I(A2: A A1
(21)
Similarly, apply Lemma B. decompose the second term in equation 78
1 [[BB;A)) - + B2(Ai) I(B;; B2) + (BBi A1)
1(Bi B2)
(22)
End of proof.
B.2 MUTUAL INFORMATION AND SUPERVISD LEARNING TASKS
A core objective of machine learning is to learn effective data representations. Many methods attempt to To achieve this goal through maximizing mutual information (MI), e.g. InfoMax principle (Linsker) 1988) and information bottleneck principle (Tisbby et al., 2000). Unfortunately, estimating MI is intractable in general (McAllester & Stratos 2020). Therefore, many works resort to optimize the upper or lower bound of MI (Alemi al. 2016 Poole al. 2019 Ni et al. 2022)
In the field of self-supervised learning (SSL), there are two widely used methods for acquiring meaningful representations: contrastive methods and predictive (generative) methods. Recently, has been discovered that these two methods are closely linked to the maximization of lower-bound
mutual information (MI) targets. A summary of these relationships is presented below.
15

---

Published as a conference paper at ICLR 2024
B.2.1 CONTRASTIVE OBJECTIVE
Contrastive learning (CL) [Chen al. 2020a) learn representations that are similar between positive pairs while distinct between negative parss. From the perspective of mutual information maximization, CL actually maximizes the mutual information between the representations of positive pairs. The
InfoNCE loss (Oord et al., 2018: Kong et al. 2019) is given by:
f(x, 3)
LInfONCE = ppp((.))) log
(23)
Ejes (x,
where (2r,3) is positive pair, Y is the sample set containing the positive sample y and [})]
negative samples of x, ((::)) characterizes the similarity between the two input variables. (Oord
al., 2018) proved that minimizing the InfoNCE loss is maximizing lower bound of the following
mutual information:
I(X;Y) Z log D
LInfoNCE-
(24)
Denote and 122 as two views of the input and ha is the representation function. Define x = ho(v1)
and y = (v2))) as representations of the two views and the similarity function f(x,)) = exp(x y). contrastive learning is optimizing the following InfoNCE loss (Arora et al. 2019)
exp(ho(v1) ha(w)))
LeL -Eperr 25) [log
(ho(m)))))) + Eng exp(ho Tho())))
(25)
where uzt is the positive sample, 12 is negative samples. Accordingly, minimizing the CL. loss is maximizing the lower bound of / (1) ho w.t.. the representation function.
B.2.2 PREDICTIVE OBJECTIVE (MASK- MASK-THEE THEN-HREDICC
The mask-then- teee-thhe--rreictt task (Devlin et al.) 2018) revealed to maximize the mutual information between the representations the context and the masked tokens (Kong et al. 2019). lower bound of this MI can be derived in the form of predictive loss:
T(X;Y)= H(Y) H(Y H((IX)
(26)
Ep(z.g) logp(u|>)] 2 Eptca) log [(vlx)].
The last inequation holds by applying the Jensen inequation Ep(ra) log ((vlz) log Ep(eav) p(ul=).
g(ulz)) plyl
Denote x = (c) and y = (m) as representations of the context e and the masked token m to
be predicted. qa the predictive model. This predictive objective Epte,n) [log qoi (hg (m) (ha()))
corresponds to the training objective of a mask-then-predict task. Therefore, according to equation 26
task maximizes the lower bound of the MI between representations of the context
and the masked tokens, i.c.
hoM)) 2 Ep(c.)) log (h))mm))))
(27)
B.2.3 GENERATIVE OBJECTIVE
(Liu et al. 2022b) conducts cross modal pretraining by generating representations of one modality from the other. Utilizing equation 26 and the symmetry of mutual information, we can derive lower bound of MI in the form of a mutual generative loss:
I(X; Y) Epce.))) ((y])))))) + log q<xly)).
(28)
Denote ii and V2 as two views of the input. hy is the representation function and go is the predictive model. In equation 28 let ho(v)) and ho (v2), then we can derive that learning to generate the representation of one view from the other corresponds to maximize the lower bound of mutual
information between the representations of the two views:
I(ho)V)), (o 2 Ep(m) log qo (he (vn (v1)))o((z)) + log ge,
(29)
16

---

Published as a conference paper at ICLR 2024
B.2.4 MODALITY BLENDING
We next present an theoretical understanding of multimodal blend-then-predict, For simplicity, we consider two relations, denoted as R2D = (ai))xxx and RaD (baj)nxn. Their elements are randomly partitioned into two parts by random partition variable S, represented as RID A2], R3p [1.. B2], such that A, shares identical elements indexes with B,, i {1,2]. The blended matrix denoted as RRDASD = [A1; B2]. Our objective to predict the two full modalities
from the blended relations:
max Eg Epplaa,. .2.b. b2) log Qon 0.01 db2
+ log 402 (ha(())))))) (42))))) (30)
where hg is the representation extractor, 9ő, and log are predictive head that recovers R2D and R3D- Utilizing the result from equation] 27 the blend-then-predie objective aims maximize the lower bound of mutual information presented below:
Es (he (2);; he (A1), ) hy (Bz)) + 1 ( ha he (B2)).
(31)
From the mutual information decomposition in Proposition B.1 the objective in equation 31 can be divided into two parts.
Est F B1) I(A B2) + [(A;; B2) + I(A2: B21 I(A2:
comtrastive and generative
conditional contrastive and generative
(32)
(2) + (BB; + A2) B2
mask: predict
multimodal mask then predict
The first part of Equation 32 corresponds to existing (conditional) contrastive and generative methods, which aim to maximize the mutual information between two corresponding parts (A; with B, i 21) across two modalities. The second part represents the (multimodal) maas--then-predee objectives. focusing maximizing the mutual information between the masked and the remaining parts within a single modality.
This decomposition demonstrates that objective unifies contrastive, generative (inter-modality prediction), and mask- predict (intra-modality prediction) approaches within a single cohesive -reddct framework, from the perspective of mutual information maximization. Moreover, this approach fosters enhanced cross-modal interaction by introducing an innovative multimodal mask-then-prediin target.
C EXPERIMENTAL DETAILS
C.1 DATASETS DETAILS
MoleculeNet (Wu et al. 2017) 11 datasets are used to evaluate model performance on 2D tasks:
The blood-brain barrier penetration dataset, aims at modeling and predicting the barrier permeability.
BBBP:
Tox21:
This dataset ("Toxicology in the 21st Century")) contains qualitative toxicity mea- surements for 8014 compounds on 12 different targets, including nuclear receptors and stress response pathways.
ToxCast is another data collection providing toxicology data for large library of compounds based on in vitro high-throughput screening, including qualitative results of over 600 experiments on 8615 compounds.
ToxCast:
SIDER: The Side Effect Resource (SIDER) database of marketed drugs and adverse drug reactions (ADR), grouped into 27 system organ classes.
ClinTox: The ClinTox dataset compares drugs approved by the FDA and drugs that have failed clinical trials for toxicity reasons. The dataset includes two classification tasks for 1491 drug compounds with known chemical structures: (1) clinical trial toxicity (or absence of toxicity) and (2) FDA approval status.
17

---

Published as a conference paper at ICLR 2024
Table 8: Hyperparameters setup for pretraining.
Hyperparameter
Value
Max learning rate
le-5
Min learning rate
Learning rate schedule
cosine
Optimizer
Adam
Adam betas
(0.9, 0.999)
Batch size
4096
Training steps
1,000,000
Warmup steps
100,000
Weight Decay
0.0
num. of layers num. of attention heads embedding dim num. of 3D Gaussian kernel
12
768
128
Table 9: Search space for MoleculeNet tasks. Small datasets: BBBP, BACE, ClinTox, Tox21, Toxcast, SIDER, ESOL FreeSolv, Lipo. Large datasets: MUV.
Hyperparameter
Small
Large
HIV
Learning rate
[1e-6, le-4] (32,64.128,256)
[1e-6, le-4]
[1e6, lc-4] (128,256)
Batch size
(128,256)
Epochs
(40, 60, 80, 100)
(20, 40)
(2 10)
Weight Decay
[lc-7, lc-3]
[1e-7, le-3]
[le-7, 1c-3]
The Maximum Unbiased Validation (MUV) group is another benchmark dataset selected from PubChem BioAssay by applying a refined nearest neighbor analysis, con- taining 17 challenging tasks for around 90,000 compounds and is specifically designed for validation of virtual screening techniques.
MUV:
HIV: The HIV dataset was introduced by the Drug Therapeutics Program (DTP) AIDS An- tiviral Screen, which tested the ability to inhibit HIV replication for over 40,000 compounds.
BACE: The BACE dataset provides qualitative binding results for a set of inhibitors of human f-secretase 1522 compounds with their 2D structures and binary labels are collected, built as a classification task.
ESOL:ESOL a small dataset consisting of water solubility data for 1128 compounds.
FreeSolv: The Free Solvation Database provides experimental and calculated hydration free energy of small molecules in water.
Lipo: Lipophilicity is an important feature of drug molecules that affects both membrane permeability and solubility. This dataset provides experimental results of octanol/water distribution coefficient (logD at pH 7.4) of 4200 compounds.
QM9 RRamakrishnan al.) 2014 QM9 is a quantum chemistry benchmark consisting of 134k
stable small organic molecules, corresponding to the subset of all 133,885 species out of the GDB-17 chemical universe of 166 billion organic molecules. The molecules QM9 contains up to 9 heavy atoms. Each molecule is associated with 12 targets covering its geometric, energetic, electronic, and thermodynamic properties, which are calculated by density functional theory (DFT).
C.2 HYPERPARAMETERS
Hyperparameters for pretraining and finetuning on MoleculeNet and QM9 benchmarks are presented in Table X Table and Table TO repectively.
18

---

Published as a conference paper at ICLR 2024
Table 10: Hyperparameters for QM9 finetuning.
Hyperparameter
QM9
Peak Learning rate
le-4
End Learning rate
le-9
Batch size
128
Warmup Steps
60,000
Max Steps
600,000
Weight Decay
0.0
Table 11: Ablation studies on fintuning settings of 2D tasks.
Finetuning Settings
BBBP
Tox21
ToxCast
ClinTox
Bace
ESOL +
FreeSolv
Lipo +
2D
73.0
77.8
66.1
87.6
83.7
0.831
1.910
0.638
2D + 3D
71.8
76.8
67.4
90.9
84.3
0.874
1.824
0.636
D ABLATION STUDIES
D.1 2D TASKS WITH 3D INFORMATION
Since our model pretrained to predict both 2D and 3D information, for 2D tasks, we consider utilizing the 3D information predicted by our model supplementary information (2D + 3D in Table IT}. We observe that both settings achieve comparable performance across various tasks. This may be due the 2D and 3D spaces have been well aligned and 3D knowledge is implicit injected into the model, allowing to achieve satisfactory results even with only 2D information provided.
19


Reviews:
[{"review_id": "bnKGSvn6Ch", "reviewer": "Reviewer_XaRC", "summary": "This paper introduces a molecular representation learning method that fusing the information from both 2D and 3D molecule structures. A unified relation matrix is constructed to describe the relationships between each pair of atoms, so that both 2D and 3D information can be injected into the matrix for fusion. For the 2D structure, based the bonds between atoms, shortest path and edge type information can be calculated for each entry of the relation matrix, and for the 3D structure, the entry can records the 3D Euclidean distance between atoms. The 2D and 3D information are blended in one relation matrix and a Transformer backbone is trained to recover the full information.", "questions": "1. Does the author run all baseline methods on the experimented splits or cite some results from other papers? For the results in Table 4, does the  single-modality mask-then-predict strategies use the same network as the proposed blending strategy?\n\n2. In Table 4, I think the author compares blending three mask to using only one mask. What's the effect of blending two masks?\n\n3. The conclusion in section D.1 is not well supported, since different 3D networks may perform differently and may have different speed of convergence. It's not enough to draw the conclusion by comparing the proposed method to only one 3D model.\n\n4. Why run ablation studies on different datasets and different tasks?", "limitations": "", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The idea of using a relation matrix to unify the 2D and 3D information for molecular representation learning is novel.\n2. Theoretical analysis provide more insights to the proposed method.\n3. The paper is well writen and structured and easy to follow.", "weaknesses": "1. The information gathered in the relation matrix is quite limited and much information in the original structure is lost, especially those in the 3D structure. The matrix construction is quite similar to the work of  \"One transformer can understand both 2d & 3d molecular data\" published in ICLR 2023.\n2. Ablation studies on blending two masks should be provided.\n3. Some details of the experimental setup is missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hB3Pb5HJCI", "reviewer": "Reviewer_TmmD", "summary": "This paper proposes MolBlend, a method that explores the intrinsic alignment between 2D and 3D for molecule pretraining. MolBlend aims to conduct the 2D-3D pretraining based on the atom relations, which is finer-grained than previous works.", "questions": "- I am confused about the Fig 1.b. Does this mean the input can be either 2D or 3D, and the output can be both 2D and 3D?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The key idea is clear and straightforward: to use the attention module to help augment the 2D-3D atom-relation for molecule pretraining.\n- The theoretical proof is interesting.", "weaknesses": "- The motivations are not clearly claimed or supported.\n    - For instance, on Page 2, the authors say that they “observe that although appearing visually distinct … are intrinsically equivalent as they are essentially different manifestations of the same atoms and their relationships”. What does “equivalent” mean here? A lot of 2D-3D pretraining methods start by saying such two modalities are complementary to each other.\n    - Additionally, on Page 2, what is the motivation to feed both modalities as one unified data structure to one single model in MoleBlend?\n\n- Notations are misleading in Sec 3.1.\n    - Why is $R_{spd}$ required? Because they can be derived from $R_{edge}$?\n    - For Eq 1, the notation should be $R_{2D3D,S}$ (with S in the subscript).\n    - Is $R_{2D3D}$ the masked version of $R_{spd}, R_{edge}, and R_{distance}$?\n\n- Other minor comments:\n    - The title can be further improved, especially that what modalities are considered is not explicit.\n    - The distance modeling is invariant. A more advanced equivariant model is preferred here.\n    - It would be better to explicitly add a column in the result tables on what backbone models are pretrained/comparing.\n    - The citation of 3D InfoGraph is wrong. Please fix it.\n\nI will consider raising the score once the authors answer/fix these comments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xzf6XJZSNi", "reviewer": "Reviewer_w2xw", "summary": "The authors proposed to align molecule 2D and 3D modalities at the atomic-relation level and introduce MOLEBLEND. This multimodal molecular pretraining method explicitly utilizes the intrinsic correlations between 2D and 3D representations in pertaining. Extensive evaluation demonstrates that MOLEBLEND achieves state-of-the-art performance over diverse 2D and 3D tasks, verifying the effectiveness of relation-level alignment.", "questions": "Please see the weakness.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written and easy to understand.\n2. The authors conducted extensive experiments on both 2D and 3D molecule tasks and showed their good performance.", "weaknesses": "1. The authors did not discuss the training time cost of different pretraining methods.\n2. A series of pertaining baselines are missed in related works and comparisons. For example:\n\n[1] Xu M, Wang H, Ni B, et al. Self-supervised graph-level representation learning with local and global structure. ICML 21.  \n[2] Zhang Z, Liu Q, Wang H, et al. Motif-based graph self-supervised learning for molecular property prediction. NeurIPS 21.  \n[3] Zaidi S, Schaarschmidt M, Martens J, et al. Pre-training via denoising for molecular property prediction. ICLR 23.  \n\n3. The theoretical analysis is good. However, could the authors provide more insights into why MOLBLEND overperforms existing methods theoretically?\n\n4. How does the different choice of encodings for 2D/3D modalities influence the pretaining?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nMnOEKHWcY", "reviewer": "Reviewer_3A6t", "summary": "This paper seeks to understand the inherent connection between 2D and 3D representations, capturing the essential structural attributes of molecules through atomic-relation level multimodal pretraining techniques. \nIn this process, the authors initially combine atom relations from various modalities into a single cohesive matrix for combined encoding, and subsequently retrieve specific information for both 2D and 3D structures separately.", "questions": "Provided above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Learning the qualified representation of molecules is important for various downstream tasks.\n2. To the best of my knowledge, this is the first work that aligns atom-level representation for 2D and 3D representations of molecules.", "weaknesses": "1. As highlighted by the authors, earlier studies typically aligned different modalities at a broader molecule level, potentially hindering the capture of detailed molecular structures. Initially, when considering atom-level depictions of a molecule, our focus might be on the atom-level rather than the atomic-relation level. For instance, we might design a model that determines 3D atom coordinates based on a 2D molecular graph. What's the rationale behind emphasizing atomic-relation level multi-modal pretraining? It would be beneficial to provide a thorough reasoning in the methodology section and draw empirical comparisons in the experiments.\n\n2. The experimental findings appear to be underwhelming. MoleBlend's performance, as shown in Tables 1 and 2, doesn't seem to fare well against prior studies.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)