Structure-Aware Fusion with Progressive Injection for Multimodal Molecular Representation Learning
Authors: Zihao Jing
Yan Sun
Yan Yi Li
Sugitha Janarthanan
Alana Deng
Pingzhao Hu
Venue: NeurIPS 2025 poster ()
Keywords: Multimodal Learning
Molecular Property Prediction
Structured Fusion
Cross-Attention
Mamba
3D Robustness
Low-Resource Pretraining
Primary area: deep_learning
TL;DR: We propose MuMo, a structured fusion model with progressive injection that improves robustness to 3D noise and modality collapse..

Abstract: Multimodal molecular models often suffer from 3D conformer unreliability and modality collapse, limiting their robustness and generalization. We propose MuMo, a structured multimodal fusion framework that addresses these challenges in molecular representation through two key strategies. To reduce the instability of conformer-dependent fusion, we design a Structured Fusion Pipeline (SFP) that combines 2D topology and 3D geometry into a unified and stable structural prior. To mitigate modality collapse caused by naive fusion, we introduce a Progressive Injection (PI) mechanism that asymmetrically integrates this prior into the sequence stream, preserving modality-specific modeling while enabling cross-modal enrichment. Built on a state space backbone, MuMo supports long-range dependency modeling and robust information propagation. Across 29 benchmark tasks from Therapeutics Data Commons (TDC) and MoleculeNet, MuMo achieves an average improvement of 2.7% over the best-performing baseline on each task, ranking first on 22 of them, including a 27% improvement on the LD50 task. These results validate its robustness to 3D conformer noise and the effectiveness of multimodal fusion in molecular representation. The code is available at: github.com/selmiss/MuMo.

Structure-Aware Fusion with Progressive Injection for Multimodal Molecular Representation Learning
Zihao Jing', Yan Sun', Yan Yi Li', Sugitha Janarthanan', Alana Deng', Pingzhao Hu!
'Deaarmeent of Computer Science, Westem University, London, ON, Canada DDepartment of Biochemistry, Western University, London, ON, Canada zjing296uwo.com phu490uwo.
Abstract
Multimodal molecular models often suffer from 3D conformer unreliability and modality collapse, limiting their robustmess and generalization. We propose MuMo, structured multimodal fusion framework that addresses these challenges in molecular representation through two key strategies. To reduce the instability of conformerr-deenddnt fusion, we design Structured Fusion Pipeline (SFP) that combines 2D topology and 3D geometry into unified and stable structural prior. To mitigate modality collapse caused by naive fusion, we introduce Progressive Injection (PI) mechanism that asymmetrically integrates this prior into the sequence stream, preserving modality-speciff modeling while enabling cross- modal enrich- ment. Built on state space backbone, MuMo supports long-range dependency modeling and robust information propagation. Across 29 benchmark tasks from Therapeutics Data Commons (TDC) and MoleculeNet, MuMo achieves average improvement of 2.7% over the best best-performing baseline on each task, ranking first 22 of them, including 27% improvement on the LD50 task. These results vali- date its robustness to 3D conformer noise and the effectiveness of multimodal fusion in molecular representation. The code is available github.com/eeliss//Muu..
Introduction
Molecular property prediction is fundamental computational chemistry and drug discovery, offering cost-effective alternative to experimental screening. According to the Tufts Center for the Study of Drug Development, developing a new drug costs over $2.6 billion on average, with much this attributed to early-stage trial inefficiencies [Chatterjee 2015
Accurate silico prediction substantially reduces time and cost by early elimination of suboptimal candidates |Graff et al. 2021]. To improve prediction, recent advances in molecular representation learning have explored large-scale pretraining or multimodal architectures, typically combining SMILES [Weininger] 1988) 2D graphs, and 3D geometries. Sequence-
models [Fabian et al. 2020 Ross et al. 2022 leverage mature language modeling but often miss structural detail, while 3D-aware models Stärk al. 2022 capture geometric context at the cost of scalability and stability. These limitations highlight the necessity for an efficient and structure-awar fusion framework.
Specifically, we identify two key challenges in molecular representation learning: (1) Conformer- dependent fusion is unreliable. First, conformers generated by tools like RDKit often differ significantly in local arrangement even with the same molecule. As shown in Figure a), two RDKit- generated conformers exhibit clear geometric differences in the rotation and orientation terminal groups, despite sharing identical 2D topology and SMILES string. These conformers may present
Corresponding author. Department of Computer Science, Western University, 1400 Western Road, London, Ontario N6G 2V4, Canada. E-mail: phu49 @uwo.ca (P.H.)
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

different surface areas or spatial constraints, leading to changes in predicted properties [Adams and Coley, 2025 Brethomé al. 2019]. Second, some different molecules share nearly identical embeddings, making them difficult to distinguish. Figure Tb) illustrates this conformer sensitivity using two drugs (lbuprofen and Ketoprofen). Despite being chemically distinct, their conformer embeddings from DimeNet [Gasteiger et al.] [020] exhibit considerable overlap in Principal Compo- nent Analysis (PCA) space, indicating the risk that existing embedding methods fail to distinguish between structurally similar yet functionally distinct molecules due to conformational noise.
(2) Modality collapse stems from naive fusion. In many multimodal models, different modalities are treated as equally important and are fused in the same phase using simple operations such as early concatenation or token-level attention. This is based on an untenable assumption that all modal- ities are clean and semantically aligned. However, in molecular data, 3D inputs are often noisy, and different modalities (e.g., geometry and SMILES) operate at distinct levels of abstraction. These can lead to modality collapse. where the 3D signal dom- inates or distorts the information from other modal- ities (Su et al., 2020 Li et al.] 2020]. Prior stud- ies in vision- lanion- and chemistry [Rong al. 2020, ZZeg et al. 2023] also observe that symmet- ric fusion often leads to unstable optimization degraded generalization. These findings inspire a shift toward asymmetric fusion, allowing for pre- cise and properly timed information exchange be-
(a) Two conformers show local 3D variation
7.5
5.0
6.0
-25
-3.5
Truprofen
-10
Kateprofan
PCI
(b) Conformers reveals PCA embedding instability
Figure 1: Illustration of Limitations in molecu- lar representation learning.
tween modalities.
To address these challenges, we propose MuMo, a multimodal fusion model for molecular repre- sentation learning with two key components. To mitigate the unreliability of conformer-dependen fusion, we introduce a Structured Fusion Pipeline (SFP) that combines 2D and 3D inputs, as well as local and global information, into unified and aligned graph representation. It serves as stable structural prior for the subsequent inference. To mitigate modality collapse from naive fusion, we propose Progressive Injection (PI) mechanism that asymmetrically integrates the fused structural prior into the main sequence stream, while preserving the independent propagation and evolution of the modality information throughout the model to grasp the long-range dependencies in complex molecules, Together, these enable MuMo to model molecules into a consistent representation in a robust and structure-an aware manner. We summarize our key contributions as follows:
We propose a Structured Fusion Pipeline that aligns and encodes 2D and 3D inputs into a unified and stable structural prior, addressing the inconsistency of conformer-dependen modeling.
We introduce a Progressive Injection mechanism that asymmetrically integrates structural prior into the mainstream, mitigating modality collapse caused by inappropriate fused 3D signals.
Our MuMo ranks top on 22 out of 29 molecular tasks across fusion baselines, 3D-heavy models, and larger pretrained models, averagely outperforming previous methods by 2.7%, even up to 27% on the LD50 dataset, showing the leading practical value in molecular property prediction.
2 Related Work
Single-modality molecular models. Sequence-based models such as MolBERT Fabian et al. 2020). MoLFormer [Ross et al.) 2022), and CRemBERTa [Chithrananda et al. 2020 treat SMILES ES as language and leverage Transformer pretraining, but lose structural fidelity due to serialization. Graph neural networks like GCN Kipf and Welling 2016), HiGNN [Zhu et al. 2022 and FPGNN (Cai et al.) 2022] preserve 2D connectivity, capturing local atomic patterns but lacking geometric awareness. 3D models, including SchNet (Schhütt et al. 2018), DimeNet [Gasteiger et al.) 2020), and Uni- Mol (Zhou et al. | [023), encode spatial coordinates and bond angles, but depend on force feeld-eriied conformers OOliveira et al. 2020), being expensive and fragile for large and flexible molecules.

---

Lniified Graph Representation
(e) Fusion Pipeline and Progressive Injoction
SMILES
Gate Ciuphi Netwerk (Frbbediing)
Tokemizer
Aliess with Festures
Emibedding
Chemical Bonde with Foxurrze
Sequence Feature
MPNN
MPNN
Geumany Fesure Betwvars Bends
Topology
Mamba
MPAN
Gand Graph Fasing Natwortk wewe
3D Geocnetry Peoture
Mamba
Indi Tiphue
into Ujekee
Faraad Nost
Suquease
Sequetze
Fasico Direction
MartMaa
2D Topology Streama
Glaed
Aln Mini with Faric
VEA
Feskzo Updat
(b) Geomelry awwre Substructurc Partitioning
Maltinedll 1B
Attn-Mamba
Gated Graph Fusion Network
BRXCS
MMIT
Att--Mamta
IEA
Gited Gruph Fision Network
Optminer
Seeak
Altn-Mamaa
Injeetion-Fnhancool
Traemaation
Molocular
Representation
Figure 2: Overview of the MuMo architecture. (a) Structural Unified Representation for 2D/3D modalities encoding, (b) Substructure Partitioning for multiscale molecular feature, (c) Fusion Pipeline (right) of 2D topology & 3D geometric priors, and Progressive Injection to integrate cross- modal structural information into the main sequence (left).
Multimodal and pretrained models. Several models explore the fusion of multiple molecular modalities. GraphMVP [Liu et al. 2022] and MolCLR (Wang et al. [222]] combine 2D graphs with 3D conformers using contrastive pretraining. while Uni-Mol [Zhou al. 2023] integrates topological and geometric features via coordinate-awe adopt symmetric fusion strategies, which can entangle noisy 3D signals and suffer from conformer perturbation. Separately, large-scale pretrained models such ChemBERTa [Chithrananda et al.) 2020). TranFoxMol [Gao et al. 2023), and MLM-FG [Peng al. 2024) focus on SMILES-only inputs, improving sequence modeling through masked prediction or fragment-awand encoding. However, they lack explicit mechanisms for integrating structural priors, and their performance is often tied to scale rather than architectural robustness. We also discussed the connection between reused modules and our contributions in Appendix B. and B.2
graph encoders. These models typically
3 MuMo: Structured Fusion and Progressive Injection
We present MuMo, molecular representation framework comprising two core components. Sec- tion 3.1 provides an overview of the MuMo architecture. Section 3.2 and 3.3 introduce the Struc- tured Fusion Pipeline and the Progressive Injection mechanism, which address the inconsistency of conformer-dependent fusion and the modality collapse caused by naive integration.
3.1 Overview of MuMo Architecture
The fusion pipeline of MuMo consists of two key pathways, as illustrated in Figure (1) Structural modality fusion stream (Section (32)): The 2D molecular graph (purple lines) and 3D geometric information (green lines) are jointly fused to generate unified structural representations, which evolve through independent propagation. (2) Semantic sequence stream (Section 3.3): The SMILES sequence is first tokenized and embedded, being input into the stacked modeling blocks as the main stream (blue line). And the structural modality information stream will be injected into the main stream at later layers for subsequent inference, and generate the ultimate molecular representation.
During inference, MuMo introduces the progressive injection mechanism to selectively integrate the structural priors from the modal fusion stream to the main sequence stream. Unlike symmetric fusion strategies, the structural priors are treated as auxiliary guidance and are asymmetrically injected via dedicated attention layers (purple dashed arrows). This allows the sequence to first establish its own contextual semantics before receiving structural guidance, avoiding signal distortion while effectively

---

Sub Unified T.
SMILES
Sub Bend liddens bg
SNGR
Downerram Tooks
Sequence Llddeens
Propetics Prediction
Suh Read Hiddens N
Iliddens
My
Gased Pasion
Drspo-Atziiinn
Molecuiar Sinilary
Glomnry Hidklnn
hg
Dond Friddess
NNGRW
Asom Hiddens
fty
Furber Tadka
Unified Graph T
Globbal Pooling
InjoctoonEnhanes
Structured Fusion Pipeline
Enhanced
Figure 3: Multimodal Fusion. It illustrates how structural modalities (2I) and 3D) are fused via the Structured Fusion Pipeline (SFP), then injected into the SMILES sequence stream through the Injection Enhanced Attention (IEA, within PI) module.
enriching the multimodal structural information. The final molecular representation is derived from the structurally enhanced sequence output.
3.2 Structured Fusion Pipeline for 2D and 3D Modalities Integration
To establish a unified molecular structure, we propose a graph-based representation that seamlessly integrates both 2D typological, 3D geometric information at both local and global scales in the graph, serving as a joint structural prior for guiding sequential semantic modeling.
3.2.1 Structural Unified Graph Representation for 2D and 3D Modalities
To jointly represent and process multimodal molecular data, MuMo introduces Unified Graph formulation, a two-step message passing mechanism, and a batch-aware graph aggregation scheme.
To better integrate molecular information in 2D and 3D modalities, define a Unified Graph structure T (V, V. G), where V denotes atoms (nodes), chemical bonds (edges), and G auxiliary geometric linkages. As illustrated in Figure aa), each atom v; E V is associated with a feature vector v; E Rdv and each bond eij E & with eij E Rds. The geometric linkage set G encodes spatial relations between edge pairs (eij. ex) sharing central atom v,. represented as a triplet 8ija = (1j. Ja. Oiax), where I, = llp: pylla is the Euclidean bond length and Oij the angle between bond vectors. To guarantee geometric consistency across conformations, we formally prove the rotational invariance of the proposed representation in Appendix A.T
Unified Graph Structure:
Message Updating in the Unified Graphs: As Figure (left) shows, to jointly propagate 2D and 3D geometric information, we perform two-step message updating throughout the graph. At each iteration f, edge-centered and node- centered updates are computed as follows, here ())
Algorithm 1 Unified Batching Scheme
T T-batch) +- new(T), 8, < 0, 8e + Vi.....TTh Output: (batch
Input: List of Unified Graphs
1:
for = 1 to N do
hey, h and hg.. denote the hidden states of edges, nodes, and geometric descriptors, respec-
2:
1"t (1) Merge Entity Features //
3:
Vibatcch - Pibatch) U UT.VV
tively:
4:
E(botch) @batch) UT:E8
5:
Glbatch) - (butch)) UTxG
m ann
Σ
Message"" 1) h R (1)
6:
1 (2) Adjust Constraints #/
NgENeteu)
7:
Cdhatec)) C
(bbutcc))
h way = Update of & (h
1 m (1))).
T Ty.Csl
Cg + -c ( U((dx(ee))+b, (dxlew
(batck)
8:
m
Σ
Message (1e)) 11 h
(3)
6.) I [Idxlen), [Ixxee)).dxx(ell E Tx
vyeNyyth)
9:
1" (3) Update Offsets "/
Update"((h)) m (+))).
(4)
10:
6, - S, + V...V de - 6, + IT
end for
the final node embeddings h, are subsequently
12: Batch Idx: T (hotch)) Idx - [k- 1 1 N
passed to the sequence stream via the PI (Pro-
13: return (batch)
gressive Injection) detailed in Section 3.3.1
To enable batch processing of Unified Graph, we adopt unified batching scheme that merges N Unified Graphs (T...... TNl into single batched graph (utch)) as outlined in
Unified Batching Scheme:

---

Algorithm This process proceeds in three stages: (1) merging entity features (e.g., node and edge attributes) from individual graphs. (2) adjusting constraint sets, including topological and geometric linkages, to ensure consistency across graphs, and (3) updating node and edge index offsets to preserve intra-graph referential integrity. This process aggregates constraint sets into global representations. The resulting batched graph supports vectorized message passing while maintaining the structural integrity of each constituent molecule. Given 6 Å distance graph, multi-layer message passing allows implicit reconstruction of dihedral angles (Appendix(A..
3.2.2 Geometry-Awar Substructure Partitioning for Multiscale Representation
Conventional molecular segmentation typically relies on 2D topology or SMILES sequences, often neglecting the rich spatial information in 3D conformers. To address this, adopt geometry-awan substructure partitioning strategy that incorporates 3D cues to refine molecular decomposition.
Graph Partitioning: We extend Breaking of Retrosynthetically Interesting Chemical Substructures (BRICS) rules [Seo et al. 2023) (details Appendix (B.2) to spatial graphs by severing a subset of topological edges Ecar Eglobal identified from the fused Unified Graph T global
(V, Eglobal 9 Gllobal). We retain the remaining connectivity and angular constraints to construct a segmented graph T sub composed of batch of N disconnected subgraphs.
Suu =
Eglobal \ &eeus> sab
=globall \
Batch( 1) T sub-
(5)
Message Updating in Multiscale Graphs: To extract multiscale structural representations, we perform message passing over both the global graph Tglobal and the segmented graph sub. The resulting node embeddings hy from the global view and huy from the substructures are then combined via a gated fusion scheme. This fusion adaptively balances coarse-grained global semantics and fine- grae-gra local features:
B = a (y (concat(hyy, hry by suy) h = B bv + (1 -B) (concat(hyy, hv
where ep(+) and (6-) are learnable transformations.
We combine multiscale structural signals into unified structural prior for subsequent injection. Specifically, the fused node representations hy are aggre- gated into global structural prior representation, which encapsulates both global and local structural features. This structural prior is subsequently used to enhance the semantic modeling stream via PI (Progressive Injection) in Section 3.3
Aggregation of Unified Structural Prior:
3.3 Progressive Injection for Structural Prior and Sequence Fusion
then integrate it into the sequence stream without disrupting the dominant modality-specific semantics. To this end, we introduce the Progressive Injection (PI) asymmetrically inject structural information into designated token rather than performing complete fusion. Next, the Structural Prior Evolution mechanism propagates the structural information independently across layers to enable a high-level semantic awareness.
Given the fused structural prior in Section 3.2
3.3.1 Injection Enhanced Attention for Structural Priors and Sequence Stream
As shown in 3 (right), injection Enhanced Attention (IEA) performs as the core module PI in each fusion layer, which integrates structural priors into the SMILES sequence stream through three sequential operations: priors extraction, sequence and structure alignment, and global semantic injection. then
Step 1: Extract structural priors from the Unified Graph. As shown in Algorithm 2 we begin by extracting node embeddings h from the batched graph T (batef), and unbatch them into per-graph node features (i) (line 1-2), which serve structural priors. In parallel, the SMILES sequence is represented as h for subsequent semantic modeling.
Step 2: Align sequence and structure via cross-attention. We first apply self-attention over h to capture intra-sequence context (line 5), then perform bidirectional cross-aftention: structure attends to sequence (line 6), and sequence attends to structure (line 7). The updated structural representation h (++1) is rebatched into graph format 4
(++1) (line 8).
5

---

Step 3: Inject structural prior into the global anchor token. To inject the fused structure into the sequence stream, we aggre- gate (++1) into pooled prior hepoled (line 9), which is added to the [GTK] (global token) via a residual update (line 10). The updated graph (ha(hechh)) is then passed forward (line 11) for independent evolution for higher-level perception, which we discuss in Section 3.3.2
Algorithm Injection Enhanced Attention
Input: Sequence hiddens fty .graph T Output: h g
f Step 1: Prior Extraction $/
1: h - ( .V
he Unbatch(hh... ( T-batch)) .batch)
3: Øs. Ks. VsS c- Linear(h (1)
F. Kr. VF t Linear(h (p))
We adopt a staged modeling strategy that sep- arates initial sequence encoding from cross- modal integration. In the early layers, the Mamba backbone operates exclusively on SMILES tokens, capturing intrinsic sequence- level semantics without external interference. In the latter part of the model, progres- sively inject the fused structural prior into the sequence stream. This delayed injection al- lows the model to establish stable token rep- resentations before being modulated by struc-
1 Step2: Sequence and Structure Alignment */
5: h < SelfAttention(() Qs. Ks. Vs))
h h < CrossAttention Qr. Ks. Vs)
7: (++l) - CrosssAttention Qs. Kr, Vy)
h < Batch(h (6+1) gr(batch) bbtch)
1" Step3: Prior Injection "/
9: pooled GlobalPool(hh.. (++1) )(bcch
10: h (++))
SIGTKI - Norm(h (++1)
S.GGKI aahe pooled,
11: Tharch)) V - hay
tural information, preserving modality auton-
12: return h (++1) T (batch)
omy and improving convergence.
3.3.2 Structural Prior Evolution by State Space Propagation
To further enhance the global perception of structural representations, we propose evolution strategy based on state space modeling. Inspired by the well-known characteristic of neural networks that shallow layers typically capture local textures while deeper layers learn higher-level semantics, we allow the fused 2D and modal signals to propagate independently across network layers.
We adopt Mamba blocks as the backbone for temporal consistency and continuous evolution of structural priors through state-space dynamics. Unlike Transformers, which rely on layerwise token-to-token attention, Mamba maintains recurrent latent state that evolves across layers. This architecture naturally accommodates injected priors gd regulating the state trajectories driven by inputs without directly disrupting the interactions among local tokens. Consequently, structural priors are seamlessly integrated throughout the semantic modeling stream. At each layer 7, we maintain latent state (0 the saat-space update at each layer is:
(++1) = Az B,h B.g", h = C21+)) +D
(
where h is the sequence state, g0 is the structural prior, and A, B,, By. C, are learnable parameters. Our evolution scheme by latent recurrence allows structural priors persist and influence downstream layers in a controlled, interpretable manner. Therefore, gradually increases the receptive field and enables progressive abstraction of structural features.
4 Experiments & Downstream Analysis
In this section, we conduct comprehensive experiments to evaluate the performance, robustness, and consistency MuMo across diverse molecular tasks. pretrained MuMo on the ChEMBL-11.6M dataset [Gaulton et al. 2012 via masked language modeling (MLM), followed by task-specific fine-tuning (see Appendix C.4 In addition, we present ablation studies and visualization analysis to show the contribution of each component in enhancing the multimodal integration and improving the overall quality of molecular prediction. Extended experiments and ablation studies can be found in Appendix and Appendix respectively.
4.1 Datasets and Baselines
Datasets. To evaluate performance and generalization ability, we benchmark MuMo on 29 tasks from three widely used platforms: 14 from the TDC [Huang et al. 2021), which provides rigorous

---

Table 1: Results on selected benchmark tasks from TDC and MoleculeNet. We report AUROC for classification (1) and MAE/RMSE for regression (L) tasks. We provide the results of "mean.." over runs. The top 2 scores per task are highlighted in pink. "Tox-Avg" and "CYP-Avg" indicate average AUROC over (DILI, hERG, Ames) and (CYP2C9-I, CYP2D6-I, CYP3A4-I), respectively, Notably, MolBERT does not natively support multi-objective tasks (SIDER, TOX21).
MODELS
BBB
HIA
PGr
BLON:
Tox-AvG.
CYP-AvG.
Tor2Cxn/10
TDC DATASETS CLASSIFICATION AUROC1
AFPENIVEF
0.855o.011
0.974a.07
0.892oo12
0.632n.039
0.842o.010
0.749
FPGNN
0.888
0,958
0,930u.007
0.6660.03
0.860u017
0.8666004
DMPNN
0.8640.010
0.9760.001
0.8890.005
0.6170.050
0.8210.019
0.819..004
ATTR MASKING
0.892n.022
0.978
0.929
0.5770.057
0.8460.01
7.
CONTEXTPRED
0.8977.004
0.975au04
0.923ux05
0.67 16.22
0.818
0.8276.003
MoL
0.8680.019
0,9510.006
0,875o.011
0.6199.019
0.8376.017
0.8600.006
0
DEREMOL
0.774m.03
0.880
0.82
0.5099.036
0.735u.01
0.770
MuMo
0.8999.044
0.979k003
0.9429.019
0.7140.021
0.8400.015
0.880.007
MODELS
BACE-R
BACE-S
BBBP-R
BBBP-S
CLINTOX
SIDER
TOX21
MOLECUENEE CLASSIFICATION AUROC 1
FPGNN
0.831o.011
0.831
0,9040.020
0.8920.019
0.732x.068
0.6610.014
0,8330.004
Mou
0.780o.012
0.780
0.9076.024
0.8810.015
0.8300.047
0.6360
0.8160.011
CHEMBERTA-2
0,8480.007
0.8480.07
0,9320.037
0,8922.019
0.9330.054
0.7080.00
0.8099.029
MoLFFRRRR
0.8730.09
0.833
0.889b.028
0.868
0.888
0.651o016
0.804
M0LBERT
0.8820.015
0.832
0.9550.008
0,9496.013
0.8750.041
GROVER
0.779o.059
0.779
0.8496.008
0.8230.020
0.6850.066
0.6350.034
0. 0.808
Usi-MoL
0,840o.031
0.840.051
0.889b.025
0.886
0.818
0.6666.021
0.812
MuMo
0.8780.066
0.8 0.8490004
0.962
0.9570.011
0.9850.011
0.6776.009
0.8340.009
MODELS
LD50
Caco-2
PPBR
LIPO
MODELS
ESOL
FREESOLY
TDC DAIASESS
REGRESSION
MAE J
Mouscura REOREsSION
RMSE
APTENTVEFP
0.678no12
0.4010.02
9.3730.335
0.5720.007
CHIMBERTA-2
0.633o.132
1.2199.206
FPGNN
0.638
0.3260
8.4651
0.544o.011
FPGNN
0.658o
1.1060.95
DMPNN
0,6076.02
0.388
8.1580314
0,4480.014
GROVER
0.617a077
1.9010.459
ArTRMASKING
0.6850.00
0.5460.052
10.0750.202
0.5470.024
MoLFORMER
0,653o.029
1.1900.046
CONTEXTPRED
0.669u.c0
0.502
9,445a.224
0.535u.012
MoLBERT
0,6170.091
1.31
TRANFoxMoL
0.6450.006
0.487a.0s
9.0550.523
0.5250.024
TRANFoxMoL
0.9300.261
1.2250.155
DineMoL.
0.5899.006
0.327
9.5330 162
0.6606.004
Uoi-MoL.
0.7690
1.5980.15
MuMo
0.4260.001
0.315
7.3
0.448
MuMo
0.536o
1.0820.085
Table 2: Evaluation QM9 benchmarks from Uni-Mol-v2 [T et al. 2024 Results MAE (L).
Standard errors are in gray subscript. The top and second top results are highlightad in pink.
MODEL
HOMO/LUMO/GAP L
a 1
C,
1
# L
R²
ZPVE
GROVER BASE
0.0079
2.3650.30
1.1030.330
0.618
113.014.06
0.0035
GROVER-LARE
0.0083
2.2400.85
0.8533.186
0.623
85.85
0.0038 s..4
GEM
0.0067tt:05
0,58900042
0,237o.0137
0.444..015
25.670.743
0.001 22
UNI-MoL
0.00432:05
0.3630.009
0.1833.002
0.155
4.8050.055
0.001 35
M 310M
0.00361::05
0.3150003
0.1433.002
0.092a.003
4.672x.245
0,0005
UNi-Mor2 570M
0.0036:--05
0.3
0.1477.007
0.089
4.5236..80
0.0005
UNI-Mor2 1.1B
0.0035 16-05
0.305 003
0. 0.1440.002
0.089t.00m
4.2656.067
0.0005
MuMo 505M
0.0030
0.28 003
0.1066.0
0.400
18.08h.533
0.0005
absorption, distribution, metabolism, excretion, and toxicity (ADMET) challenges and leaderboard baselines, and 12 from MoleculeNet [Wu et al. [018). along with 3 chemical tasks from Reaxtica [Ln ct al.) 2022] which enables evaluation against strong unimodal and pretrained models. These tasks cover a range of molecular properties, including bioactivity and ADMET-related endpoints, for both classification and regression.
We compare MuMo against various competitive baselines spanning diverse modalities and pretrained algorithms. These include 3D-aware models like FPGNN Cai al. 2022] and Uni-Mol [7hou et al. 2023] that incorporate spatial geometry, 2D graph-based models such as HiGNN et al. [021] and GCN (Kipf and Welling 2016) that rely solely on molecular topology, and pretrained models include GROVER [Rong et al. 2020]. MoLFormer [Ross et al. 2022 and
Baselines.
7

---

Table 3: Evaluation on catalytic activity and reaction yield benchmarks from Reaxtica Lin et al. [022]. MuMo achieves the best performance on three tasks. Standard deviations are shown gray subscript where available. The best result is highlighted.
BHC (R' 1, ReaCTION YIEL)
CPA (MAE 1, CATALYTIC AcTIvITY)
HTE (Ré 1, REACTION YIELD)
MODELS
VALUE
MODELS
VALUE
MODELS
VALUE
REAXTICA
0.94
REAKTICA
0.144
REAXTICA
0.87
MFF
0.92
MFF
0.144
RXNFP
0.81
RXNEP
0.95
DENMARK ET AL.
0.152
DRFP
0.85
MuMo
0.952
MuMo
0.144
MuMo
0.873v.002
Table 4: Ablation study on the effectiveness of Structural Fusion Pipeline. "2D" column refers the use of either 2D topological information or SMILES sequence alone. Mean and standard deviation reported for two classification tasks (AUROC) and regression tasks (RMSE).
CLASSIFICATION
REGRESSION
2D
SUG
GSP
BACE 7
BBBP
ESOL
LIPO
IMPACT
V
V
V
0.849
0.957
0.536mm
0.577
0.00%
V
x
V
0.82 Incoe
0.956mas
0.664ams
0.615
-7.46%
V
0.849
0.960
0.585
0.614
-3.00%
X
X
0.84
0.949ka0
0.654
0.630
-7.29%
X
0.766
0.956
0.719
0.655
-13.11%
2022 that requires large-scale self-supervised learning before fine- tuning on specific tasks. The comprehensive comparisons forcefully demonstrate the effectiveness of MuMo in molecule representation across architectures, input modalities, and learning paradigms.
ChemBERTa-2 [Ahmad et al.
Settings. We follow official protocols or recommendations for fair comparison in each bench- mark. AUROC is used for classification; MAE (TDC) and RMSE (MoleculeNet) for regression. MoleculeNet tasks use scaffold split for single-objective classification; otherwise, random. Each task is run 5 times: we use the official leaderboard splits for TDC and generate 5 splits for Molecu- leNet (Train:Valid:In)) Valid:Test=8:1:)) Hyperparameters follow each baseline's s official setup or defaults unspecified. Additional details about datasets and settings provided in Appendix C.55
4.2 Main Performance and Analysis
As Table [ shows, MuMo outperforms the best baseline by an average of 2.7% across 21 bench- mark tasks from TDC and MoleculeNet, rank- ing first 17 of them, and improves up to 27% on LD50 compared to DeepMol [Cor- other fusion models like Uni-Mol [Zhou et al.] [2023] and TranFoxMol [ao al. | [2023], MuMo exhibits more consistent performance across different tasks, validating the benefit of progressive and asymmetric integration of structural information. On conformer-sensitiee regression tasks such as PPBR and LD50, MuMo maintains the lowest eтror. highlighting its robustness to geometric noise.
Table 5: Evaluation QM7/8 and QM9 (HO- MO/LUMO//APP benchmarks from MoleculeNet. Results are MAE (L). Standard deviations are in gray subscript. The top result is highlighted.
reia et al.
2024 Compared
MODEL
QM7 1
QM8 1
QM9
GROVER
92.0m9
0.0224a.0m
0.0099
DMPNN
103.5a6
0.0190
0.008 0.00000
ATTENTIVEF
72.02.
0.0179p.0mo
0.0081 0.0000
UNIMOL
41.80.2
0.01560.000
0.0047o.0000
McMo
42.8
0.011 10.000
0.0030
As shown Table 2 and MuMo consistently outperforms other baselines on QM7/8/9 datasets (7 out of 10 tasks), which are known to be sensitive to conformer and molecular geometry, further highlighting robustness conformer-sensittee and superior 3D molecular modeling capability.
4.3 Broader Chemical Benchmarks
Our original design focuses on single-molecule property prediction, which is why the baselines and benchmarks were selected accordingly. However, we also investigate the model's generaliza- tion to broader chemical domains beyond individual molecules. To this end, extend MuMo

---

1.6
1.6
SMIL15 O3l-
SMILES Craph
0.5
SMILLS
0.5
SMILLES-Gagfre Gcoeedry
1.2
1.2
R-4
0.8
0.8
8.3
0.3
0.4
0.2
0.2
2000
2000
5000
8000
2000
4000
6000
4000
2000
4000
1000
3000
2000
40.0
6000
8000
Training Steps
Figure 4: Pretraining loss curves under different modality configurations. Each part shows training (left two figures) and validation (right two figures) loss for a pairwise modality comparison.
to accept reaction-level inputs and evaluate it on four datasets from Reaxtica Lin et al. (0022). following their official data splits and reported baselines. These datasets cover two important do- mains: catalytic activity and reaction yield. As shown in Table MuMo achieves the best results on three out of four tasks, surpassing prior methods such as Reaxtica, MFF, rxnfp, and DRFP.
4.4 Ablation Studies
Contribution of components in Structural Fu- sion Pipeline. We conduct ablation studies on MoleculeNet tasks assess the effect of core components in SFP: "SUG" (structural unified graph representation for multimodal signals), and "GSP" (geometry-aware substructure par- titioning). As shown in Table using only se- quence information results in the largest degra- dation (-13.11% 11),, which shows the importance of leveraging both topological and geometric sig- nals. Removing 3D geometry (no SUG) leads to a significant drop (-7.46%), and removing the GSP, which aggregates the local and global structural features, results in -3.00%. These re- sults demonstrate not only the effectiveness of aligning 3D information into unified represen- tation and encapsulating multi-scale structural signals, but also highlight that each component provides complementary benefits that essential for the robustness of SFP. In particular, the synergy between SUG and GSP allows the model to capture richer chemical priors, yielding stronger generalization across diverse molecular benchmarks.
Table 6: Impact of injection and its timing. Results on BBBP (AUROC) and ESOL (RMSE). "@a-b"
denotes injection between layers and b.
VARIANT
BBBP 1
ESOL J ARoP
MuMo 9-16
0.9576.011
0.536
0.00%
FULL-IN INI @ 1-16
0.954..002
0.587o.025
-1.85%
EARLY-INJ @ 1-8
0.9466.0s
0.597u.022
-2.96%
LATE-INI @ 13-16
0.96 100002
0.6176.028
-4.40%
NONE-INI (Seo)
0.928m.044
0.939
-18.91% %
Table 7: Impact of injection approaches (progres- sive injection vs. fixed injection).
VARIANE
BBBP t
ESOL
AvG.DRoP
PROGRESSIVE-IN 0.957amm
0.536
0.00%
FIXED-INI 0.9466.0s
0.597
-6.28%
Effects of components in Progressive Injection. To evaluate the effectiveness of our injection strategy and the independent propagation of the structural prior, we conduct two ablation studies in T and 7 From Table [ structural prior injection improves the performance by wide margin regardless of timing (14.51%) %, 15.95% and 17.06% for late//early-ffull injection). However, early-/full- injection introduces modality collapse due to underdeveloped sequence semantics, while late-injection provides inadequate structural information. Our MuMo injects from layer 9, yielding the best results by balancing semantic establishment and structural guidance. As for propagation (Table 7). using the fixed structural prior throughout the inference will hinder the progressive refinement of structural representations. Fortunately, by propagating structural prior, MuMo improves 6.28%, demonstrating the benefits of the independent evolution of structural information across layers.
Impact of input modalities on pretraining dynamics. Figure illustrates the impact of structural signals on pretraining loss curves. Compared to SMILES- only, adding 2D graph consistently accelerates convergence and lowers both training and validation loss, indicating that topological priors enhance early learning. Further incorporating geometry leads to the lowest loss across all steps, suggesting that spatial information provides strong inductive signals for alignment. These trends highlight the complementary role of each modality in guiding effective pertaining.

---

Layer 2-8
10 Scaffolds
Layer 10-16
Figure 5: Layer-wise representation of the pretrained model. UMAP of embeddings across 10 selected scaffolds (5,000 molecules), showing scaffold-level separation different layers.
4.5 Visualization Insights
To assess the effectiveness of generated embeddings in distinguish- ing distinct molecules, we analyze their Pearson correlation with established molecular dissimilarity or similarity representations. Specifically, we randomly sample 20,000 molecules from the ZINC et al., 2018) to construct molecule pairs. We then generate their embed- dings by MolFormer and MuMo, and compute the embedding distance for each pair. We measure the Pearson correlation between the embedding distances and the distances from widely used struc- tural metrics: Tanimoto distance and MCS substructure overlap. As shown Figure 66 MuMo exhibits stronger correlations than MoLFormer, demonstrating its ability to produce robust molecular embeddings and reflect underlying structural relationships. See Appendix C.7 for details.
Molecular Similarity Analysis.
dataset
Representation Distribution with Structural Prior. To investigate how structural priors af- feet the evolution of molecular representations across network layers, we perform layer- wise analysis of embeddings using Manifold Ap- proximation and Projection (UMAP). As shown in Figure S in the early layers before injec- tion (Layers 1-8), scaffold separation gradually emerges, indicating that the model is progres- sively extracting semantic features from the se- quence stream. In the later layers (Layers 9-12), where structural priors are injected, the distri- butions become more compact and form clear scaffold-specific clusters. This indicates that the structural prior reinforces global perception without disrupting the semantic patterns learned before. This validates our motivation for progressive injection: establishing sufficient modality- specific encoding before introducing cross-modal guidance for discriminative representations.
Tanimuto Distonce: (V-axis)
MCS Nurt Atoms (Y anis)
E65
Foait Denilly
M1Maao
5
15
3.08
Molluuerr
0.6
Buclidean EX axdss
Figure 6: Visualization of similarity analysis of two models: MuMo and MoLFormer.
5 Conclusion
We introduce MuMo, a structured multimodal framework designed to address the unreliability of conformer-dependeen fusion and the modality collapse caused by naive modality integration. MuMo includes the Structured Fusion Pipeline, which combines 2D topological and 3D geometric information into stable structural prior, redueing sensitivity noisy inconsistent conformers. Progressive Injection (PI) mechanism then asymmetrically injects the structural prior into the sequence stream and evolves independently, enabling cross-man moss-l enrichment while preserving semantic autonomy. Extensive experiments on a wide range of tasks show that MuMo consistently performs over various baselines and robust on conformer-sensse multimodal approach for building robust, geometry-awar molecular models. While MuMo is currently tailored to a aaek-specific model, future work will focus on extending into general- purpose multimodal backbone for molecular representation learning.
tasks. It highlights MuMo as a promising
10

---

Acknowledgments and Disclosure of Funding
This work was supported in part by the Canada Research Chairs Tier II Program (CRC-20211- 00482), the Canadian Institutes of Health Research (PLL 185683, PJT 190272, PJT204042), the Natural Sciences, Engineering Research Council Canada (RGPIN-2021-04077 and The Canada Foundation for Innovation (CFI) John R. Evans Leaders Fund (JELF) program (#43481).
References
Keir Adams and Connor W. Coley. The impact of conformer quality on learned representations of molecular conformer ensembles. arXiv preprint arXiv:2502.13220 2025.
Walid Ahmad, Elana Simon, Seyone Chithrananda, Gabriel Grand, and Bharath Ramsundar. Chemberta-2: Towards chemical foundation models. arXiv preprint arXiv:2209.01712, 2022.
Simon Axelrod and Rafael Gomez-Bombarelli. Molecular machine learning with conformer ensem- bles. Machine Learning: Science and Technology, 4(3):035025, 2023.
Andreas Bender and Robert C Glen. Molecular similarity: a key technique in molecular informatics.
Organic & biomolecular chemistry, 2(22)::204-3218 2004.
Alexandre V. Brethomé, Stephen Fletcher, and Robert S. Paton. Conformational effeets on physical- organic descriptors: The case of sterimol steric parameters. ACS Catalysis, 9(3):2313 13-2231-221 13-233,,,,,, 2019.
Hanxuan Cai, Huimin Zhang, Duancheng Zhao, Jingxing Wu, and Ling Wang. Fp-gnn: versatile deep learning architecture for enhanced molecular property prediction. Briefings in bioinformatics, 23(6):bbac408, 2022.
Lena Chatterjee. Tufts study on drug development costs: How our pharmaceutical company-funded center can influence drug prices. TuftScope, page 29, 2015.
Seyone Chithrananda, Gabriel Grand, and Bharath Ramsundar. Chemberta: large-scale self- supervised pretraining for molecular property prediction. arXiv preprint Nxii::0000009885, 2020.
Joao Correia, Joao Capela, and Miguel Rocha. Deepmol: An automated machine and deep learning framework for computational chemistry. biorxiv. bioRxiv, 2024.
Sopanant Datta and Taweetham Limpanuparb. Steric effects electron delocalization: new look into the stability of diastereomers, conformers and constitutional isomers. RSC advances, 11(34): 20691-20700, 2021.
Benedek Fabian, Thomas Edlich, Héléna Gaspar, Marwin Segler, Joshua Meyers, Marco Fiscato, and Mohamed Ahmed. Molecular representation learning with language models and domain-relevant auxiliary tasks. arXiv preprint arXiv:201 1.1230,, 2020.
Jian Gao, Zheyuan Shen, Yufeng Xie, Jialiang Lu, Yang Lu, Sikang Chen, Qingyu Bian, Yue Guo, Liteng Shen, Jian Wu, et al. Transfoxmol: predicting molecular property with focused attention. Briefings in Bioinformatics, 24(5)bbbad306, 2023.
Johannes Gasteiger, Janek GroB, and Stephan Günnemann. Directional message passing for molecular graphs. arXiv preprint arXiv:2003.03/23 2020.
Anna Gaulton, Louisa J Bellis, A Patricia Bento, Jon Chambers, Mark Davies, Anne Hersey, Yvnnn Light. Shaun McGlinchey, David Michalovich, Bissan Al-Lazikani, al. Chembl: a large-scale bioactivity database for drug discovery. Nucleic acids research,
100--11107, 2012.
Rafael Gómez-Bombarelll, Jennifer N Wei, David Duvenaud, José Miguel Hernández- Lobatoo Benjamín Mancceez-Lenglling Dennis Sheberla, Jorge Aguilera-Iparraguire Timothy D Hirzel, Ryan P Adams, and Alán Aspuru-Guzik. Automatic chemical design using data-driven continuous representation of molecules. ACS central science, 4(2):268-22 276. 2018.
11

---

David E Graff, Eugene Shakhnovich, and Connor W Coley. Accelerating high-th nightthroughput virtual screening through molecular pool-based active learning. Chemical science, 12(22):7866-788) 2021.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023.
Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitmik, Percy Liang, Vijay Pande, and Jure Leskovec. Strategies for pre-training graph neural networks. In International Conference on Learning Representations (ICLR), 2020a.
Weihua Hu, Bowen Liu, et al. Strategies for pre-training graph neural networks. In ICLR, 2020b.
Kexin Huang, Tianfan Fu, Wenhao Gao, Yue Zhao, Yusuf Roohani, Jure Leskovec, Connor W Coley, Cao Xiao, Jimeng Sun, and Marinka Zitnik. Therapeuties data commons: Machine learning datasets and tasks for drug discovery and development. Proceedings of Neural Information Processing Systems, NeurlPS Datasets and Benchmarks, 2021.
Xiaohong Ji, Zhen Wang, Zhifeng Gao, Hang Zheng, Linfeng Zhang, Guolin Ke, et al. Uni-mol2: Exploring molecular pretraining model at scale. arXiv preprint arXiv:2406. 14969, 2024.
Sunghwan Kim, Paul A Thiessen, Evan E Bolton, Jie Chen, Gang Fu, Asta Gindulyte, Lianyi Han, Jane He, Siqian He, Benjamin A Shoemaker, al. Pubchem substance and compound databases. Nucleic acids research, 44(D1): D1213, 2016.
Thomas N Kipf and Max Welling, Semi-superve supervised classification with graph convolutional networks. arXiv preprint (609:66000020000 2016
James Law, Zsolt Zsoldos, Aniko Simon, Darryl Reid, Yang Liu, Sing Yoong Khew, A Peter Johnson, Sarah Major, Robert A Wade, and Howard Y Ando. Route designer: retrosynthetic analysis tool utilizing automated retrosynthetic rule generation. Journal of chemical information and modeling, 4963)2593-602, 2009.
Xiaowei Li et al. Oscar: Object-semantics aligned pretraining for vision-language tasks. ECCV, 2020.
Yujia Li, Yue Li, et al. Hignn: hierarchical graph neural network for learning molecular representa- tions. arXiv preprint arXiv:2106.05408, 2021.
Kaixian Lin, Jian Li, Hao Lin, Jianfeng Pei, and Luhua Lai. Reaxtica: knowledge-guided machine learning platform for fast and accurate reaction selectivity and yield prediction. ChemRxiv, 2022. doi: 10.26434/chemrxxvv ht/////hhmxxi--22222 9mmm-9
Shengchao Liu, Hanchen Wang, Weiyang Liu, Joan Lasenby, Hongyu Guo, and Jian Tang. Pre- training molecular graph representation with 3d geometry. In International Conference Learning Representations (ICLR), 2022.
Simon C Lovell, J Michael Word, Jane S Richardson, and David C Richardson. The penultimate rotamer library. Proteins: Structure, Function, and Bioinformatics, 40(3):389-408, 2000.
Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint
2018
Kurt Mislow and Jay Siegel. Stereoisomerism and local chirality. Journal of the American Chemical Society, (06(1)):3319-332) 1984.
Harry L Morgan. The generation of a unique machine description for chemical structures-a technique developed at chemical abstracts service. Journal of chemical documentation, 5(2):107-113, 1965.
Marina Oliveira, Maurice Andrey, Salome R Rieder, Leyla Kern, David Hahn, Sereina Riniker, Bruno AC Horta, and Philippe H Hunenberger. Systematic optimization of a fragment-basee force field against experimental pure-liquid properties considering large compound families: Application to saturated haloalkanes. Journal of chemical theory and computation, 16(12):7525-7555 2020.
12

---

Tianhao Peng, Yuchen Li, Xuhong Li, Jiang Bian. Zeke Xie, Ning Sui, Shahid Mumtaz, Yanwu Xu, Linghe Kong, and Haoyi Xiong. Pre-trained molecular language models with random functional group masking. arXiv preprint arXiv:2411.01401, 2024.
John W Raymond and Peter Willett. Maximum common subgraph isomorphism algorithms for the matching of chemical structures. Journal of computer-aided molecular design, 16:521-5333 2002.
David Rogers and Mathew Hahn. Extended-cconeettvvn fingerprints. Journal of chemical information and modeling, 50(5):742- -2474422744. 2010.
Yu Rong et al. Self-supervised graph transformer on large-scale molecular data. NeurIPS, 2020.
Jerret Ross, Brian Belgodere, Vijil Chenthamarkkshan, Inkit Padhi, Youssef Mrouch, and Payel Das. Large-scale chemical language representations capture molecular structure and properties. Nature Machine Intelligence, 4(12): 126::12566-11 12644 2022.
Ansgar Schuffenhauer. Peter Ertl, Silvio Roggo Stefan Wetzel, Marcus A Koch, and Herbert Waldmann. The scaffold tree- visualization of the scaffold universe by hierarchical scaffold classification. Journal of chemical information and modeling, 47(1))47-58, 2007.
Kristof T Schütt, Huziel E Sauceda, P-J Kindermans, Alexandre Tkatchenko, and K-R Müller. Schnet-a deep learning architecture for molecules and materials. The Journal of Chemical Physics, 148(24), 2018.
Seonghwan Seo, Jaechang Lim, and Woo Youn Kim. Molecular generative model via retrosyntheti- cally prepared chemical building block assembly. Advanced Science, 10(8):2206774, 2023.
Silas W Smith. Chiral toxicology: it's the same thing... only different. Toxicological sciences, 110 (1):4-0, 30, 2009.
Hannes Stärk, Dominique Beaini, Gabriele Corso, Prudencio Tossou, Christian Dallago, Stephan Günnemann, and Pietro Liò. 3d infomax improves gnns for molecular property prediction. In International Conference on Machine Learning, pages 20479-20502. PMLR, 2022.
Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, and Jiaya Wei. Vlbert: Pretraining of generic visual-linguistic representations. ICLR, 2020.
Yuyang Wang, Jianren Wang, Zhonglin Cao, and Amir Barati Farimani. Molecular contrastive learning of representations via graph neural networks. Nature Machine Intelligence, 4:279-287 287, 2022. doi: 00.1038/44225-0022000444
David Weininger. Smiles, a chemical language and information system. 1. introduction to methodol- ogy and encoding rules. Journal of chemical information and computer sciences, 28(1):31-36, 1988.
Peter Willett, John M Barnard, and Geoffrey M Downs. Chemical similarity searching. Journal of chemical information and computer sciences, 38(6):983-996, 1998.
Zhenqin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh S Pappu, Karl Leswing, and Vijay Pande. Moleculenet: benchmark for molecular machine learning.
Chemical science, 9()):513-530. 2018
Zhaoping Xiong, Hongming Wang, et al. Pushing the boundaries of molecular representation for drug discovery with the graph attention mechanism. J. Med. Chem., 2019.
Kevin Yang, Linus Swanson, Wengong Jin, et al. Analyzing learned molecular representations for property prediction. JMLR, 2019.
Liang Zeng, Lanqing Li, and Jian Li. Molkd: Distilling mross-modal knowledge in chemical reactions for molecular property prediction. arXiv preprint arXiv:2305.01912 2023.
Gengmo Zhou, Zhifeng Gao, Qiankun Ding, Hang Zheng, Hongteng Xu, Zhewei Wei, Linfeng Zhang, and Guolin Ke. Uni-mol: A universal 3d molecular representation learning framework. In International Conference on Learning Representations (ICLR), 2023.
Weimin Zhu, Yi Zhang, Duancheng Zhao, Jianrong Xu, and Ling Wang. Hignn: A hierarchical informative graph neural network for molecular property prediction equipped with feature-wise attention. Journal of Chemical Information and Modeling, 63(1):43-55, 2022.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction accurately summarize our two main contribu- tions-SFP and Pl-and their motivation. We verified consistency with the methods and experimental results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discussed the limitations and future direction in the Appendix B33
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model cell-specfficatinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
14

---

Answer: [Yes]
Justification: We have proved the geometric completeness of our Unified Graph representa- tion in Appendix A which meets the requirements.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We have discussed the implementation and experiment details Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. .., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
15

---

reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
We have provided the anonymous Github code link in the abstract, which includes running instructions, anonymous datasets, and pertaining checkpoints download links, as well as the training logs of each downstream dataset.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
public/guides//ooeSuiie ottpaddseecdeettmmsssssooooooo. stt.bmmmisssooooocccoo
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permirted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have described details in both the Section 4 and the Appendix C We also provide all the reproducing scripts in the code.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We provided the error bars for each experimental result in this paper. See Section 4 and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
16

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτоr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We have discussed the details about training in Appendix C.4 for pertaining and Appendix C.5.2 in finetuning.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: The research complies with the NeurlPS Code of Ethics. It uses public datasets, involves no human or sensitive data, and poses no foreseeable ethical concerns.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., - if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: This paper discussed the broader impacts Appendix B.3
Guidelines:
The answer NA means that there is no societal impact of the work performed.
17

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deploymentt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work uses only publicly available molecular datasets and does not release any models for generating content with potential for misuse. Therefore, no special safeguards are necessary.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All datasets used in this work, including those from MoleculeNet and TDC, are publicly available under appropriate licenses and are properly cited. We do not use or modify any third-party pretrained models proprietary code.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
18

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, papprswii thcode. com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
Jatasets,paperswi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We release the pretrained weights of our MuMo model, along with instructions for loading and fine-tuning. Documentation is provided the associated code repository.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This work does not involve any crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for rescarch with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This work does not involve human subjects and therefore docs not require IRB approval equivalent review.
Guidelines:
19

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification; Large language models (LLMs) were not used part of the core methodology. Any LLM usage was limited to concept understanding and knowledge assistance and did not impact the scientific content of the paper.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non- non-standard components.
Please refer to our LLM policy for what should or should be described.
20

---

Appendix Table of Contents
A. Geometric Completeness Discussion
A.1 Proof of Rotational Invariance in the Unified Framework
A.2 Capability of the Unified Graph in Distinguishing Molecular Isomerism
A.3 Are Explicit Torsion Angles Necessary?
B. Relationship with Previous Methods
B. Mamba State Space Model
B.2 Breaking of Retrosynthetically Interesting Chemical Substructures
B.3 Limitations and Broader Impact
Implementation & Experiment Details
C.I Unified Graph Batching
C.2 Enpection-Enhanee Attention (IEA) Implementation
Substructure-LLest Tokenizer
C.4 MuMo Pretraining
C.5 Molecular Properties Prediction
C.6
Insights of Representation Learning in Pretraining
Molecular Similarity Analysis
C.8
Attention Analysis for Fusion Effectiveness
C.9 Training Cost Analysis
D. Extended Experiments and Ablation Studies
Contribution Multimodal Combinations
D.2
Pretraining Ablation Studies
D.3 Impact of Model Size
D Impact of Pooler Methods in Finetuning
D.5
Impact of Sequence Data Type
D.6
Impact of Key Modules
D.7
Impact of Number of Fusion Layers
D.8
Backbone Generalizability
D.9 Impact of Asymmetrie Fusion
D.10 Potential Data Overlap Analysis
D.II Ablation on Conformer Settings and Robustness to Geometric Perturbations
A Geometric Completeness Discussion
In this work, introduced the MuMo model, which seamlessly integrates the topological and geometric information of molecules. The accuracy and completeness of geometric representations are crucial for capturing the underlying spatial properties. In the context of molecular representation, the ability to distinguish between stereoisomers essential, as highlights the importance of capturing fine-grat fine-grained geometric detail, which is key advantage of our geometric modeling.
A.1 Proof of Rotational Invariance in the Unified Graph Structure
In this section, we present a rigorous proof that the Unified Graph structure framework, as defined by
T Emtiry (V.E, and T Tonstraiin = (Ce, Co).
exhibits invariance under arbitrary rotations in dirre-dimessional Euclidean space. We first restate the core components of the Unified Graph in concise manner, then formally define rotational invariance, and finally offer a detailed proof, complete with references to fundamental geometric identities and transformations.
Definitions. For clarity and convenience, previously defined terms in Section 3.2.1 are restated. In the Unified Graph, V is the node set, where each node v E V is endowed with feature vector
21

---

ey E Rd and spatial coordinate p; E R Edge set is &, where each edge eij E & connects the node pair (v,, vy) and is described by an edge feature vector eij E R In addition to these standard graph entities, Unified Graph introduces a geometric set & that provides essential spatial properties for each edge and its adjacent edges. Specifically, for an edge eij sharing a common vertex vj with another edge ejk, the corresponding geometric representation Bijk G is the triplet gijA (4j- Jax- O132). where
cos(O = (pi- - Pj, Pt P
1 = =<=1lp;-- Palle- -
(10)
| P/l22 op--1 Px p/l22
Here, || ll and (:.-) represent the Euclidean norm and the dot product in R respectively. The constraint group T Constraint = (Ce, Cg) codifies topological and geometric consistency via the edge index set Cg and the shared-vertex edge pair index Cg. Our focus is the invariance of (ly, LR. Biya) under arbitrary rotations.
A representation in R² is said to be rotationally invariant if, for any rotation matrix R e R that orthonormal (i.e.. RR R = I) and any translation vector b R3, the core geometric descriptors remain unchanged. Concretely, if one transforms the node coordinates via P. Rpp b, then the
resulting triplets K'y (Ki 1'x- 8 (1s)) must satisfy
L = 4j- I'x -
8F Ojja-
(11)
We next show that Unified Graph's definitions inherently guarantee this property.
Proof of Rotational Invariance. For any pair of nodes vy), consider the original coordinate difference p: Pj and its length |p:l-p Under the transformation p; Rpp + b, we have
ps pij (Rp +bb (Rp, + R(p: P(pp--P -P;).
(12)
Hence the new length is
f |pl| - pill2 = R(p: (p:: - Pi>l1 R
= (p:--p)) -pj) R R R(p:- -p:) PP))
(p: (p:) T(p: P,) = =IIp: |Ppl2 = vz). Then, the invariance of angles should be proved. We should show that U/x O,ja under the same transformation. Observe that
Since the same argument applies for Ve). we obtain 1/x Ise-
(p) p', P p')
(R(p: P.). R(p - -p))
cos(O)y)) Ipi P'ill lp' -P'12 || R p>>1l R(p= py>l|2
(13)
The numerator of this fraction can be expanded using the invariance of the dot product under orthonormal transformations:
(ot-p))(ppppp))) (R (p:-P)). R (p: = (p--p)) R R(px-p)) (p--P)) (p:-P,) (pi-P,. P Px-P)). (14)
Mcanwhile, the denominator reduces precisely to ||pp- -p/lllll 1pxl -pil/ by the argument in the proof of invariance of edge length. Consequently, we have
( R (p: Pj), R(p: p;)) ( (p: P,), (pe -P))) cos(O.x).
cos(e),.)
(15)
R -iill R (p: pi)l/ [p: 1 (p: p
Oja, ff E [0,x]
fiu- = Az
(16)
By combining the above two results, we conclude that for every edge eij and its adjacent edge ex the geometric triplet 8'ja = -Cyz //x. (. remains identical gija under any spatial rotation (and translation). Hence, the Unified Graph structure fully preserves lengths and angles, guaranteeing invariance of its geometric descriptors with respect to orthonormal transformations in R Formally. for all rotation matrices with R
and translation vectors b, the Unified Graph definitions ensure Eiye = gija. Y eij. ex E E, which completes the proof of rotational invariance in the Unified data structure framework.
22

---

A.2 Capability of the Unified Graph in Distinguishing Molecular Isomerism
The Unified Graph T proposed in this work combines topologiaa and geometric features to represent molecules. It includes the topology of nodes and edges as well as geometric descriptors G. which encode edge lengths and angles at shared vertices. This subsection evaluates the capability of the proposed representation in distinguishing different types of molecular isomerism [Axelrod and Kiomez-Bombane 2023).
Constitutional Isomers (Structural Isomers). These isomers differ in the connectivity of atoms, i... their topological structures are distinct. Since the Unified Graph explicitly encodes edge connectivity relationships in Cg. structural differences in connectivity are directly reflected in the graph, allowing effective differentiation between constitutional isomers [Datta and Limpanuparbi 2021].
Cis/Trans Isomers (Geometric Isomers, E/Z Isomers). Geometric isomers share the same connectivity but differ in the spatial arrangement of substituents due to constraints such double bonds or ring structures. These differences manifest as variations in certain interatomic distances or bond angles. The geometric descriptors ly and Oza in the Unified Graph can capture these variations, enabling differentiation between cis/trans or E/Z isomers [Smith 2009].
Diastereomers. Diastereomers, especially those with multiple chiral centers, are not mirror images and often exhibit measurable differences in local geometric features such as bond lengths, bond angles, or interatomic distances. These differences are encoded in the geometric descriptors G. allowing the Unified Graph to distinguish most diastereomers effectively.
Enantiomers (Optical Isomers). Enantiomers are -on-superinposable mirror images that are identical in connectivity, bond lengths, and bond angles but differ in their handedness. Since the Unified Graph only uses unsigned lengths and angles without encoding chirality or orientation explicitly, it cannot distinguish between enantiomers, as their representation T would be identical [Mislow and Siegel| 1984).
Conformers (Conformational Isomers). Conformational isomers arise from rotations around single bonds, typically resulting in different spatial arrangements of atoms. If these conformational changes do not significantly alter equilibrium bond lengths or angles, and if only one specific conformer is represented in the graph, such differences may not be captured. Hence, rapid interconversion between conformers is usually ignored in the Unified Graph representation [Lovell et al. 2000).
In summary, the Unified Graph T effectively distinguishes most isomer types, including constitutional isomers, geometric isomers, and many diastereomers. However, it has limitations in identifying enantiomers due to the absence of chirality-speciffc descriptors. Future extensions could incorpo- rate chirality-sensitive features, such as signed dihedral angles or higher-dimensional orientation information, enhance its capability to distinguish optical isomers.
A.3
Are Explicit Torsion Angles Necessary?
Step Are torsion angles missing? A torsion (dihedral) angle dejar is fully determined by the six inter-atomic distances of a four-atom chain (i, I):
iijs = atanz (r))) Egs (ryXrr)) (rgX)w))
where Fab X, X.. X2 (Xa. Ya. za). Since our molecular graph already provides all pairwise distances within 6 cutoff, and message passing runs at least three layers, an i-> path that closes the i- j-j-k- - quadrangle always exists. Thus, the network can implicitly reconstruct torsion angles without requiring explicit torsion features.
Why not add explicit torsion anyway? To test the benefit of explicit torsion encoding, we added [sin diju, cos d1jr]] on every rotatable bond while keeping all other hyperparameters unchanged. As shown in Table 8 the performance difference on QM9 property benchmarks is minimal: only C, shows a moderate gain (+10.)%) while HOMO/LUMO/GAP performance even drops slightly (-6.67%). However, Table shows that adding torsions sharply increases GPU memory and runtime (up to 2.6x slower), indicating that the marginal accuracy benefits not justify the computational overhead.
Step 2.
23

---

Table 8: Explicit torsion ablation on QM9. Results are MAE (L) with standard errors in gray subscript.
TAsK (MAE)
w/o TORSION
w/ TorSion
A (%)
a (L)
0,283
0.281 0,000
+0.71
C, (L)
0, 166
0.113
+ 10.3
ZPVE (J)
k.0005ke:05
0.0005
0.0
HOMO/LUMO/GAP ())
0.0030
0.0032
-6.67
AVERAGE
+1.00
Table 9: Efficiency comparison. GPU memory usage and step time are reported in Pretrain/SFT(QMB)) format with global batch size 128/64.
MODEL VARIANT
GPU MeM (GB)
STEP TIME (MS) J
OURS (IMPLICIT)
44.4/14.4 64.6/29.4
987/731 2623/1316
EXPLICIT TORSION
Step 3. Robustness to conformer sensitivity. Beyond efficiency, we further validated robustness
on QM7/8/9 conformer sensitivity benchmarks (see Section 4.. MuMo consistently outperforms UniMol under perturbations, confirming that implicit torsion modeling is sufficient and our injection- enhanced design maintains strong robustness to 3D geometric noise.
B Relationship with Previous Methods
Understanding the relationship between our proposed method and prior approaches is crucial for situating our contributions within the broader research landscape. This section aims to highlight the key distinctions and improvements introduced by model while also acknowledging the foundational principles laid by existing methodologies.
B.1
Mamba State Space Model
The Mamba Model. State Space Models (SSMs) are a class of mathematical frameworks widely used for modeling temporal sequential data by describing latent dynamics and observations Guu and Dao 2023). An SSM typically consists of two components: latent state evolution equation and an observation equation. Formally, let h, E Rd represent the latent state at time and let y, R be the corresponding observation. The SSM is defined as:
(17) where u, is input sequence, 7) and E, are process and observation noise, respectively, and A. B, C are model parameters that govern the latent dynamics and the observation process. Mamba builds on the SSM framework and introduces significant advancements enhance its efficiency and applicability to long-sequence modeling. By leveraging the SSM's inherent ability to capture long-range dependencies, Mamba employs selective scanning mechanism that optimizes the representation of sequences across diverse time scales. Specifically, Mamba avoids the pitfalls of full dense computation by introducing a structured representation of state transitions that achieves logarithmic scaling in time complexity.
1 Ah, + Bu, Th y Ch + E,,
There are several core innovations of Mamba, and that is why module for fusion modeling. a) Logarithmic Scaling in Time Complexity. Mamba reformulates the SSM's computation by leveraging selective updates to the state vector h, reducing computational overhead from ²) (where 7 is the sequence length) to O(T log 7). This efficiency makes it suitable for applications involving very long sequences, such as large molecular data genomic data.
Mamba
stacked
b) Hardware-Awwen Optimizations. Mamba introduces approximations to matrix exponentials that hurdware-friendle enabling efficient computation modern accelerators like GPUs and TPUs without sacrificing modeling accuracy.
c) General Applicability. The model supports diverse data modalities, such as text sequence, and time series, by adapting the SSM framework handle modality-speciff structures, making a versatile tool for various sequence modeling tasks.
24

---

and a nucleophile such as In Mamba, the continuous-timg time state evolution is modeled as:
dh(() = Ah(1) Bu((),
(18)
dt
where A is parameterized to ensure stability. This differential equation is solved efficiently using approximations of matrix exponentials:
h(1 + Ar) (A (
ABuBll Bu(t s)ds.
(19)
By discretizing the system with high precision and leveraging sparsity in A, Mamba achieves efficient state transitions and improved memory usage. Mamba's ' selective state-space design enables it to handle sequences spanning thousands to millions of time steps while maintaining accuracy and efficiency. These features make it particularly suitable for tasks such protein structure prediction, time-series forecasting, and for sure molecular representation learning.
Discussion of Relationship. Mamba serves as the sequence encoder in our multimodal molecular framework, offering efficient and scalable modeling of SMILES sequences via state-space dynamics. Its recurrent architecture not only improves computational efficiency but also aligns well with our Onjectio--nnhance Attention (IEA, basic module of PI) design. By maintaining an evolving latent state, Mamba naturally accommodates injected structural priors without disrupting local token interactions.
However, Mamba is not central to the methodological contributions of this work. Our core innovations, the Structured Fusion Pipeline and asymmetric cross-modal injection, define the model's 's effectiveness in robust multimodal fusion. These techniques are model-agnostic and can be applied to other sequence encoders (e.g., Transformers) Mamba's role is to enhance the stability and propagation of injected priors with sequence stream, but it does not influence the fundamental novelty or adaptability of our approach.
B.2 Breaking of Retrosynthetically Interesting Chemical Substructures
Retrosynthetic analysis is a systematic approach in organic synthesis that involves deconstructing target molecule into simpler precursor structures by breaking bonds in a logical and chemically feasible manner (Law et al. 2009]. This process is guided by the identification of strategic bonds, which, when retrosynthetically cleaved simplify the molecule while preserving its essential functional groups. The ultimate goal is to map out potential synthetic routes, starting from readily available building blocks.
In retrosynthesis, disconnection is the conceptual reversal of a formingrming reaction, often sym- arrow (=). For instance, consider the retrosynthesis of benzyl alcohol
bolized by (C&HH)-CH,OH):
CHH C,H, X-O
(20)
In this example, disconnection of the hydroxymethyl group (-CH,OH) from the benzyl group (C&Hs-) suggests two plausible precursors: benzyl halide water (HzO) or hydroxide ion (OH-).
Another classic example is the retrosynthetic analysis of aspirin (acetylsalicylic acid, CgHgO,):
CgHsO, = C7H6O; + CH,COCI
(21)
Here, the ester bond (-COO-) in aspirin retrosynthetically cleaved to yield salicylic acid (C,H,03) and acetyl chloride (CH;COCI) as precursors. These intermediates suggest a forward synthesis involving esterification:
Base
C2H,O, CH,COCI
CoHOO, HCI
(22)
The disconnection approach is not arbitrary but relies on retrosynthetic transformations that correlate to known reaction types in synthetic chemistry, such nucleophilic substitution, electrophilic
25

---

addition, or condensation reactions. By iteratively applying these transformations, a chemist can work backward from a complex molecule to identify feasible synthetic routes.
This method is particularly powerful when applied to complex natural products or pharmaceuticals, where the identification of key disconnections can dramatically simplify synthesis planning. For example, in the retrosynthesis of penicillin derivatives, the ß-lactam ring is often identified as a core structural unit to preserve, while strategic disconnections focus on assembling the side chains and core step by step.
The BRICS (Breaking of Retrosynthetically Interesting Chemical Substructures) fragmentation method deconstructs complex molecules into chemically meaningful substructures by leveraging retrosynthetic principles. Through rule-based disconnection strategies, BRICS identifies synthetically accessible bond cleavages while preserving chemically stable moieties, such as aromatic rings, and targeting bonds like carbon-carbon single bonds or carbon-heteroatom bonds near functional groups. Each fragment is annotated with a placeholder atom (e.g.. ") to mark cleavage sites, enabling recom- bination in synthetic processes. For example, benzoic acid (SMILES: CCI-CC-CC-CIC(-00) is fragmented into =CI==C=CC-CI and [*]C(=0)O, retaining the functional features of the parent molecule.
The integration of BRICS into cheminformatics tools like RDKit has streamlined its application across large molecular datasets. With automated fragmentation processes, BRICS enables the efficient generation of annotated substructures for drug discovery, combinatorial library design, and virtual screening. In fragment-basee drug discovery, BRICS facilitates the identification of minimal structural units critical for biological activity, supporting structure-activity relationship studies and lead optimization.
Discussion of Relationship. The BRICS fragmentation method plays a limited yet practical role in the molecular modeling framework presented in this work, serving primarily as a preprocessing module for extracting meaningful substructures from molecules. Its function is to provide a consistent and logical segmentation of molecular structures, supporting our geometry partitioning by instructing bond pruning. We use the instructions that describe which bond should be cut provided by BRICS to lead our geometry substructure partitioning, which is part of our innovations.
Furthermore, it crucial to emphasize that BRICS serves as an interchangeable, modular component within our workflow. While we have selected BRICS as representative example for fragment generation, the framework is designed to accommodate alternative or more advanced fragmentation techniques. This flexibility ensures that researchers can integrate methods better suited to their specific molecular systems or scientific objectives. For instance, as new chemical structures and synthesis pathways are discovered, fragmentation rules may evolve to reflect these advancements, providing an avenue for continual improvement. However, the refinement of BRICS or its alternatives not the focus of this work. Rather, our interest lies in demonstrating the versatility of our framework. allowing for the seamless substitution of fragmentation methods without affecting the validity or applicability of the overall system.
B.3 Limitations and Broader Impact
Broader Impact MuMo explores a robust and asymmetric approach to multimodal molecular fusion, aiming to improve the reliability of structure- structure-ifformeed representation learning. Beyond its immediate performance gains, the design principles behind MuMo-such as late-stage modality injection and stable structural priors-may inspire future research in multimodal learning, especially in settings where modality-specific semantics must be preserved (e.g., vision-language tasks, protein-compound modeling, or biomedical imaging). We hope our work contributes to a broader understanding of how to design more interpretable and flexible fusion strategies in deep learning.
Limitations Like most molecular learning frameworks, MuMo requires fine-tuning for each down- stream task, which can be future work, aim to develop more generalizable multitask framework based on MuMo, enabling cross-task transfer and applicability to a broader range of real-world applications in drug discovery, including candidate prioritization, toxicity screening, and multi-objective molecular optimization.
in settings with limited data or computing power. In
We emphasize that MuMo is research tool and does not provide direct clinical or regulatory advice. Responsible use of the model requires expert oversight, especially when applied to sensitive applica-
26

---

Algorithm 3 Unified Graph Batching (Detailed)
Input: List of Unified Graphs IT...... ......TT]
Output: Batched Unified Graph T-thncc) (1Vibazeh) Ethenoh) Glhunok) Cg
Clawch Chaach batch)
1: Initialize T-butchi new TO. 6, c- O, 62
Initialize offsets of nodes and edges
2: for & <- to N do
3: Step I: Merge Entity Features
4: (piharok) -Vhhaooo) UT V..VV
Merge nodes (atoms)
Elhanot Elbanch) UTE
Merge edges (bonds)
6:
g - Glonch U T2.G
Merge geometry features
7:
Step 2: Adjust Constraints
Cheatoo Chhano)) U Ó.. + (i.)) CcCCg)
Cg
Update edge index offset
9:
Chhanh
g Cg Chhuno)) u ((lxx(e,)) o,, Idx(e,)) ) + 6,) (Idx(e,;). Idx(e CgC
For geometry
10:
Step 3: Update Offsets
11:
5.
[Tt.V]
Update node offsets
12: 5, 5, (T
Update edge offsets
13: end for
14: batch e -Kk 1 k=l
Record batch index
15:
return T UNnchh)
tions such as toxicity prediction candidate drug selection. Future work may explore integrating uncertainty quantification or domain adaptation techniques to further align model predictions with safety and ethical considerations.
C
Implementation & Experiment Details
C.1 Unified Graph Batching
In the Unified Graph T, the entity group T Faiity E)) includes the node set V, edge set &, and
the geometric descriptors G. Meanwhile, the constraint group T Conntraaint = (C&, Cg) specifies topo- logical connectivity through Ca and shared- vertex edge-pair relationships through Cg. Algorithm 3 provides a procedure for merging multiple Unified Graphs IT....., TN) into a single batched graph (hotch) By appropriately offsetting the node and edge indices and unifying the constraint sets, ensures that each graph's internal structures and relationships remain consistent. Adopting such a Unified batching approach is essential when handling large-scale graph data, it facilitates parallel processing and significantly improves both training and inference efficiency.
C.2 Injection Enhanced Attention (IEA) within PI Implementation
A key challenge in multimodal learning with Unified Graphs is to effectively combine topological and geometric features with sequential embeddings. In this work, we propose an injection-enhanced attention (IEA) approach to address this challenge. As shown in Algorithm 4 after performing cross-attention between the sequence representation and the Unified Graph node embeddings, we further inject the globally aggregated features from the Unified graph into the global token [GTK] via a residual connection. This injection, modulated by learnable scalar a, enriches the [GTK] token with structural insights while preserving its original contextual content. As result, the model acquires a more holistic understanding of both semantic and geometric aspects, thereby enabling more robust information fusion for tasks that require unified representation of topology, geometry, and sequence semantics.
C.3 Substructure-Levè Tokenizer
A critical challenge when encoding molecular structures is capturing chemical nuances within the SMILES representation. To address this, we design a substructure-level tokenizer that segments SMILES strings based on chemically meaningful units (see Table[[]] Rather than splitting strictly at character boundaries, we group tokens at natural substructures such as ring closures, chirality annotations, charged atoms, multi-letter elements, and specific isotopes. This ensures that each token remains a valid chemical entity, preserving the minimal functional meaning of each fragment. Consequently, tokenizer aligns more closely with fundamental chemical principles, reduces the
27

---

Algorithm 4 Injection Enhanced Attention (Detailed)
Input: Sequence hiddens h batched Unified graph T
Output: Updated sequence hiddens h h updated Unified batch (
1: hl ( v
Extract Unified graph node hiddens
2: Step 1: Compute Queries, Keys, and Values for Cross-Attention
3: h -graph2batch_sequeeeeee sequencch h. ( .btch))
Graph sequence format
4: Qs. Kg, V/ Linear(h:))
Sequence QKV
5: QF. K5., V; +- Linear(h"))
Integrated feature QKV
6: Step 2: Perform Symmetrized Cross-Attention
7: h 1663) t- --rossAttentioont Qr. Ks. V/)
Learn from sequence hiddens
8: h [++)) <- Cross Qq. K,, VD)
Learn from fusion hiddens
9: Step 3: Injection Ennanced Feature Representation
(i 10: by < sequence2graph. & gribnich batch)
Sequence - graph format & Globul graph pooling
pooked < GlobalAddPoolinggttyy (++1) - batch) 11: by
12: hg [GTK] <- Norm(h
(++) [GTK] + h.y
Inject pooled vector into [GTK] > Update graph hiddens
13: T-ibanch V < bry (=1))
14: return f
(1+1) gr
Table 10: Pretraining hyperparameters for MuMo.
Hyperparameter
Value
Hidden size
768
Number of layers
16 (Attention-Mamba)
Number of attention heads
12
Activation function
SILU
Normalization
LayerNorm
Dropout rate
0.1 (attention)
Batch size
512
Learning rate
1 x 10
Learning rate scheduler
Cosine with 2000 warmup steps
Epochs
2
Gradient accumulation
Enabled
Precision
Mixed precision (bf16)
Training time
~5 hours on 4xA100-8000 GPUs
loss of pertinent information, and handles elaborate notations (e.g., [C@H], [12C], and %10) in chemically consistent manner. By retaining critical structural features within tokens, this approach only enhances the interpretability of token sequences but also leads improved performance across a wide range of molecular modeling tasks.
C.4 MuMo Pretraining
Pretraining Settings and Resources. The basic model setup was configured with hidden size of 768, 16 iirentiin-manblee layers, and 12 attention heads, ensuring robust model capacity. The training batch size was set to 512, with a Icarning rate of 1c-4 and a cosine learning rate scheduler featuring 2000 warmup steps. We used SILU activation inside the Mamba module, layer normalization, and dropout rates of 0.1 for both attention layers. The training spanned epochs with gradient accumulation and utilized mixed precision with bf16, optimizing computational efficiency. A single pretraining process will take around only 5 hours on 4xA100-80C GPUs.
We adopt the ChEMBL- 1.6M dataset [Gaulton et al. 2012) for pretraining, which contains curated set of bioactive molecules with experimentally validated properties. Com- pared to large-scale yet noisy corpora like ZINC (mostly synthetically accessible fragments) and PubChem (an extremely broad and noisy collection). ChEMBL provides high-quality, biologically relevant molecules that better reflect the cructure-function distributions seen in world tasks.
Pretraining Dataset.
28

---

(-), Table 11: Token categorization in the substructure-level SMILES tokenizer. Tokens are grouped by structural or semantic function, including atomic symbols, ring closures, bond types, and model- reserved tokens. Examples and definitions are provided for clarity.
CATEGORY
EXAMPLES
EXPLANATION
BASIC
C, O, S, B,
SINGLE-LETTER SYMBOLS, LOWERCASE AROMATIC FORMS. EXAMPLE, TYPICALLY DENOTES AN AROMATIC CARDON.
I, C. Mi P, B,
HAI OGENS, MULTI LETTER ELE-
CL, BR, SI, NA, CA.
TWO-LETTER SYMBOLS FOR HALDGENS (F.G., Ct.
MENTS
MG, FE, ZN, AL. K, LI. AG, SN,
BR) AND MITT-LETTER ELEMENT SYMROLS (E.G., NA, FE), OFTEN REPRESENTING METALS OR MEP-
CHIRAL/ / CHARGED
ISOTOPIC
[C@H], [co@H] [N+].
BRACKETED NOTATIONS INCORPORATING CHI-
ATOMS
[0-], [13C], [WH]
RALITY (8, (0), CHARGES (t, -), ISOTOPES
[B-]. [NA+] [Se]. [Sr@0], [NH2+],
(E.G., [13C], [14C]), AND SPECIFIC HYDROGEN
COUNTS (E.G., [NH]).
[14C]
RING CLOSURES, BRANCHING
1, 2. 3, 4, 5, 6, 7,
NUMERIC LABELS (1 9, %10, %11, ETC,) REPRE- RING
8, 9, %10 %11. C
IN MOLECULAR
BOND TYPES,
=, #,
\. 2,
VARIOUS SMILES BOND NOTATIONS:
0, 2, 5,
DOUBLE (=). TRIPLE (#), AND STEREOCHEMICAL
(/. A.. SPECIAL SYMBOLS LIKE :, ~, AND PUNC-
TUATION (S, >) ARE ALSO INCLUDED.
EXTENDED ATOMIC FORMS
[C-], [NH+]: [CH2-] J.,
VARIATIONS COMBINING CHARGE STATES ([C-),
[S-]. [x+]. [I-].
[N+]), SPECIFIC HYDROGEN COUNTS ([CH2-],
[Na] [CØ] [C08]
[NH]),
[SıH] [SN+2]. [0+].
ETED FORM.
[B-].
MORE EXOTIC ISOTOPES/RA-
[2H], [3H] [11C], [13C], [15N], [18F],
TOKENS REPRESENTING SPECIFIC ISOTOPES AND RADIONUCLIDES IN BRACKET NOTATION. THESE
DIONUCLIDES
[64Cu] [99Tc].
OFTEN APPEAR IN SPECIALIZED DATASETS, SUCH
[197Au] [238U].
SPECIAL
[GTK] [SEP]. [MASK] [UNK], [PAD], [BOS],
RESERVED
SEQUENCE PROCESSING, INCLUDING CI ASSIFICATION MARKERS, MASKS, UNKNOWN PLACEROI DERS, AND PADDING SYMBOLS.
[EOS],
This choice allows our model learn from pharmacologically meaningful signals while avoiding excessive noise or chemical redundancy.
Importantly, we deliberately pretrain relatively small molecular corpus (1.6 million molecules) and still observe fast convergence within just epochs. As demonstrated in later ablation studies (Section D.2), further scaling up pretraining data to larger datasets such full PubChem (>10M molecules) does not yield consistent downstream improvement. This finding leads to a claim: effective representation learning for molecules does not require massive-scale pretraining, especially when the pretraining set is chemically diverse and task-relevant. The MuMo model, with its efficient IEA design and structural fusion pipeline, enables strong generalization from limited-scale pertaining.
Pretraining Effectiveness. We have also done pretraining ablation studies to see how the pretraining process contributes to the downstream performance. Please see Appendix D.2 for details.
C.5
Molecular Properties Predction
C.5.1 Baselines
compare MuMo against wide range of strong baselines, categorized into three primary groups: sequence-based models, graph-based networks, and 3D geometry-aware architectures.
29

---

Table 12: Overview of datasets from MoleculeNet for molecular properties prediction experiments.
DATASET
CATEGORY
TASK TYPE
TASKS
MOLECULES
SPLIT
METRIC
BACE
BIOPHYSIGS
CLASSIFICATION
1513
SCAFFOLD
AUROC
BBBP
PHYSIOLOGY
CLASSIFICATION
1
2039
SCAFFOLD
AUROC
Tox21
PHYSIOLOGY
CLASSIFICATION
12
7831
RANDOM
AUROC
SIDER
PHYSIOLOGY
CLASSIFICATION
27
1427
RANDOM
AUROC
CLINTOx
PHYSIOLOGY
CLASSIFICATION
1478
RANDOM
AUROC
ESOL
PHYSICAL CHEMISTRY
REGRESSION
1128
RANDOM
RMSE
LIPOPHLICITY
PHYSICAL CHEMISTRY
REGRESSION
1 4200
RANDOM
RMSE
FREESOLV
PHYSICAL CHEMISTRY
REGRESSION
1
642
RANDOM
RMSE
Sequence-based models operate on SMILES strings and typically leverage Transformer-See style en-
coders trained with masked language modeling. These include ChemBERTa-2 [Ahmad et al. 2022] a RoBERTa-style model pretrained on large-scale SMILES data, and MolBERT Fabian et al. 2020 which incorporates chemically -infomeed masking strategies during pretraining. We also include MolFormer [Ross et al., 2022 a long-range Transformer architecture designed to capture global context in molecular sequences, well as TranFoxMol [ao et al. 2023 SMILES Transformer that uses -reeggrined tokenization schemes.
Graph-based models treat molecules undirected graphs, where atoms are nodes and bonds are edges. Our evaluation includes FPGNN [Cai et al., 2022 fingerprint-enhancee GNN variant), D- MPNN Yang et al. 2019]. which propagates information along directed bonds, and GROVER (Rong) cct al. 2020]. a graph Transformer pretrained with both contrastive and contextual prediction objec- lives. We also consider AttentiveFP [Xiong et al. 2019], which employs gated attention over atom neighborhoods, as well as two popular pretraining strategies from: AttrMasking (Hu et al. 2020b) (attribute prediction) and ContextPred [Hu al. 2020a (subgraph context prediction). Additionally, we include an AutoML-tuned GNN pipeline (DeepMol) [Correia et al.) (022) for automated base- line selection and optimization. For 3D geometry- aware baselines,
compare to Uni-Mol [Zhou] et al. 2023], a SEuui-equivariant molecular model that explicitly encodes atomic coordinates and interatomic distances using conformer input. It achieves strong performance structure-sensitive
tasks but is known to be susceptible to conformer variability.
All baselines are implemented using their original codebases official checkpoints when available, with training and evaluation protocols aligned ensure fair comparison.
C.5.2 Datasets & Settings
MoleculeNet Wu et al. 2018). To evaluate the performance of our method, we utilized benchmark datasets from Molecule Net, a widely recognized and authoritative resource for molecular property prediction tasks. As Table shows. the selected datasets encompass diverse range of molecular properties, covering both classification and regression tasks, as well as single-task and multi-task learning scenarios. Specifically, the classification datasets include biophysical and physiological properties, such as drug permeability across the blood-brain barrier (BBBP) and toxicity prediction (Tox21, SIDER, Clin'Tox), while the regression datasets focus on physical chemistry properties, such as solubility (ESOL) and lipophilicity.
Consistent with the official recommendation, for single-task classification problems, we adopt the scaffold split strategy, which ensures that structurally similar molecules are grouped into same subsets. This approach enhances the robustness of model evaluation by simulating realistic generalization scenarios where models must predict molccular properties for novel scaffolds. In contrast, for multi-task classification and regression problems, because multi-task models benefit from shared representations across tasks, and regression tasks often have fewer data points, making scaffold splitting too restrictive and leading to insufficient training samples, which is also not fair for the benchmarks.
random split strategy. This is
To assess model performance characteristic curve (AUROC). This metric is preferred because it insensitive class imbalance, which is common in molecular datasets. Unlike accuracy, which can be misleading when dealing with imbalanced datasets, AUROC evaluates a model's ability to distinguish between positive and negative classes across different decision thresholds. For regression tasks, the root mean square error
classification tasks, we use the area under the receiver operating
30

---

(RMSE), which quantifies the magnitude of prediction errors and is well-suited for evaluating the continuous property predictions required in physical chemistry applications.
Therapeutics Data Commons (TDC) [Huang et al., 2021]. To further evaluate the robustness and generalizability of MuMo, we include benchmark datasets from the Therapeuties Data Commons (TDC), a comprehensive collection of machine learning-ready, datasets for drug discovery and development. The selected datasets span various therapeutic tasks, including drug-target interaction (DTI), ADMET property prediction, and drug response estimation, reflecting the complexity and diversity of real-world pharmaceutical applications.
TDC provides standardized data splits and evaluation protocols, enabling reproducible benchmarking. For all included tasks, adopt the official 5-fold scaffold splits provided by TDC to ensure fair comparison across models. This split strategy partitions molecules based on their core scaffolds, ensuring that test sets contain chemically distinct compounds not seen during training. Such split is particularly challenging yet more realistic, as it simulates the practical scenario of predicting on structurally novel molecules.
For classification tasks in TDC g.. BBB, HIA), we report the area under the ROC curve (AUROC), consistent with MoleculeNet [Wu et al. 2018). For regression tasks (e.g.. LD50, Pgp), we use the mean absolute error (MAE), which is consistent with the metric on the leaderboard. All metrics are reported the mean and standard deviation across five folds to ensure statistical robustness.
Datasets Selection. We selected 14 datasets from TDC because they are part of well-curated benchmark suite with standardized leaderboards, allowing for direct comparison with state-of-the art models. These datasets focus on ADMET-related tasks, which are critical for drug development, and have been widely adopted in recent multimodal and pretrained molecular modeling studies. Where available, we include leaderboard results for competitive baselines to ensure fairness and transparency in our evaluation.
From MoleculeNet, we selected commonly used datasets for both classification and regression tasks, ensuring broad coverage of molecular properties and compatibility with prior literature. We deliberately exclude datasets such as QM9, which provide precise atomic coordinates for small molecules, as our method treats 3D geometry as auxiliary information rather than primary input. Moreover, QM9 conformers are computed via energy optimization and are known to sensitive to conformer noise an issue we aim to address rather than depend on. Our dataset choices thus prioritize both benchmarking relevance and alignment with the assumptions and goals of our framework.
Computing Resources. MuMo requires minimum of 24 GB GPU memory for fine-tuning and can be trained single NVIDIA RTX 4090. The actual training time varies by task depending on the number and size of molecules. Empirically, training on a dataset with 1000 molecules typically takes 10-20 minutes. Larger GPUs or multi-GPU setups can further accelerate training.
Hyperparameter Settings. While the overall architecture and training strategy remain consistent, some hyperparameters may vary slightly across different downstream tasks depending dataset size and task type. We do not enumerate all task-specific configurations here; please refer to our released code for full hyperparameter details and per-task settings.
C.6 Insight of Representation Learning in Pretraining
To illustrate the layer-wise feature extraction capability of our model, we present Uniform Manifold Approximation and Projection (UMAP) visualization of the hidden representations across 16 layers, shown in Figure 7 This visualization was generated by pooling the high-dimensional hidden states of molccular representations from a set of molecules and projecting them into two dimensions using UMAP [Mcinnes et al. 2018). For this analysis, selected ten distinct molecular scaffolds (Schuffenhauer et al. 2007]. each represented by a unique color in the plot to facilitate differentiation.
Scaffold (Schuffenhauer et al., 2007]. in cheminformatics, refers to the core chemical structure shared by molecules, typically including the central ring system and key functional groups. Scaffolds are widely used to group structurally similar molecules, as they often correlate with biological or chemical properties. Using scaffolds in this experiment allows us to evaluate the model's ability to capture and separate key structural features, aligning with their importance in drug discovery and molecular design.
31

---

Loyer
Liver
Loyer
Layer 4
25
34-
20-
50
ME
325
Layer
Layvr 10
Jayer 11
Layer 32
22-
60
25
190
125
83
25
20
180
125
Figure 7: UMAP visualization of the model's sequence hidden representations, where each point represents molecular scaffold and colors denote 10 distinct scaffold types. The projection illustrates progression feature separability and discriminative power across the model's 16 layers.
our experiment, the first 8 layers of the model are dedicated SMILES sequence modeling, while layers 9-16 progressively incorporate multimodal fusion through IEA. From the visualization, we observe that in the early layers (1-4), molecules from different scaffolds are mixed together, showing minimal differentiation. As the layers deepen, particularly in layers 5-8, clear boundaries begin to emerge, and molecules from
same scaffold are well-separated, indicating that the model has effectively captured the features of SMILES sequences. Starting from layer 9, topological and geometric information is gradually integrated, the scaffold separability remains stable. This analysis highlights the rationale behind introducing multimodal fusion in the latter half of the model: by first independently modeling SMILES sequences and the unified representations of topology and geometry, leverage the strengths of each modality without early-stage interference, ensuring optimal feature extraction for each data structure.
C.7 Molecular Similarity Analysis
Molecular similarity analysis evaluates the effectiveness of pre-trained embeddings (dissimilarity) by assessing their correlation with molecular dissimilarity and similarity representations. serving as a benchmark for the embeddings' scientific validity and consistency [Ross et al., 2022). For the experiments, we randomly sampled 20,000 molecules from the ZINC-250k dataset and generated 10,000 random molecule pairs. Each molecule was passed through the pre-trained model to infer its embeddings. For each molecule pair, the Euclidean distance between their embeddings was calculated and then correlated with four molecular dissimilarity and similarity representations: Tanimoto distance, the number of atoms the Maximum Common Substructure (MCS), Dice similarity, and Cosine similarity. The representations were calculated based on the Morgan fingerprint
32

---

(radius=2) of molecules, widely-used molecular descriptor that encodes structural features into fixed-length binary vectors MMorgan 1965].
To ensure the reliability of the results, we randomly divided the dataset into five groups and ran each model on these splits, reporting the mean and standard error across the five runs. We selected the ZINC-250K dataset model nor the baselines were extensively pre-trained on this dataset, ensuring fair comparison. Additionally, we chose three baseline models that are strong pre-trained models with demonstrated representation capability and robust performance on downstream molecular tasks. These baselines are well-suited for providing molecular embeddings, making them appropriate for evaluating the effectiveness of our approach. Pearson correlation was used to measure the relationship between the Euclidean embedding distance (dissimilarity) and each molecular dissimilarity or similarity representation. A strong correlation (larger absolute value of the correlation coefficient) indicates that the model's embeddings better capture the chemical similarity between molecules.
et al., [2018) for this experiment because neither our
The four representations used in this analysis provide comprehensive and scientifically validated measures of molecular dissimilarity or similarity:
(1) Tanimoto distance (dissimilarity). Widely used in cheminformatics for virtual screening and chemical similarity searches, Tanimoto distance derives from the Tanimoto similarity, fingerprint- based measure of overlap between two molecular representations. Let fa and fs be binary or real-valued fingerprint vectors for two molecules. The Tanimoto similarity T is given by:
fA fB 1l + 11 SA fB
fR)
(23)
We define the Tanimoto distance D as
D(fA- fB) 11 f8).
(24)
larger D(fA, f&) (closer to 1) indicates greater dissimilarity, whereas smaller values (close to 0) signify higher similarity [Rogers and Hahn 2010).
(2) Number of atoms in Maximum Common Substructure (similarity). Given two molecules Mol, and Mol,, the MCS identifies the largest subgraph that is isomorphic in both. We record the number of atoms in this common subgraph as:
MCS(Mol), Mol2) = MMax
(255)
where MaxSub(Mol,, Molg) the maximum common substructure (in terms of atomic count). This metric provides an interpretable measure of the structural overlap (i.c., backbone or scaffold similarity) between two molecules [Raymond and Willett 2002].
(3) Dice Similarity (similarity). Similar to the Tanimoto measure, Dice similarity emphasizes shared fingerprint features. For two fingerprint vectors fa and fa. it is defined as:
2 2(AA fa)
Dice(fa.
(26)
I| + llfall?
The Dice similarity often accentuates intersecting bits more strongly than Tanimoto, making sensitive to certain molecular distributions [Willett et al. 1998
(4) Cosine Similarity (similarity). Another vector-based metric is cosine similarity, which captures the cosine the angle between two fingerprint vectors:
fA fo
Cosine(fa. fB)
(27)
IIllllllfell
This measure remains robust to scaling, focusing on the orientation of the vectors rather than their magnitude. A higher cosine similarity indicates greater proportion of shared features between the two molecules [Bender and Glen 2004).
33

---

(9)
Tanimoto (Y-aci))
(b)
Abumt (K-asia)
(6)
Similariy (Y-428))
(d)
Similarty (Yasa))
Deciurly Noan
24
0.06
10
0.8
0.7
10
0.00
0.
Differet Aopercrtatin
05
0.0
10
10
10
Caden Srilaaty Heor
0.0
0.5
Pore Genety
20
0.4
15
0.3
1.0
0.2
622
0.3
6.1
0.0
20
30
Embedding Dachiden Distance (X 9x06))
Figure 8: Visualization of molecular similarity results of MuMo (first row) and MoLFormer (second row). The plots show the relationship between embedding Euclidean distance and four molecular dissimilarity and similarity representations: (a) Tanimoto distance; (b) number of atoms maximum common substructure (MCS); (c) Dice similarity; (d) cosine similarity for model (first row) and MoLFormer (second row). Each plot includes scatter points, and a linear regression trend line (green).
Euclidean Distance (dissimilarity). To assess the performance the model-generated embeddings, compute the Euclidean distance between the learned molecular representations (inferences from pretrained model). Given two embeddings ZA, 2B Rd the Euclidean distance is defined as:
dE(ZA, mn) ||
zall-
(28)
Since Euclidean distance measures the separation between embeddings in the latent space, larger dg value indicates greater dissimilarity, while smaller dg suggests higher molecular similarity
Correlation with Molecular Similarity Representations. To evaluate how well the embedding space aligns with established molecular similarity measures, we compute the Pearson correlation coefficient between Euclidean distance dg and four molecular similarity/diisiiilarry representations:
Tanimoto Distance D (dissimilarity):: Positively correlated with dg, as molecules with greater Tanimoto distance should have larger Euclidean distances in the embedding space.
Maximum Common Substructure (MCS) Size (similarity): Negatively correlated with dg, as molecules sharing larger common substructures should be mapped closer together.
Dice Similarity (similarity): Negatively correlated with dg, since higher Dice similarity implies molecular resemblance.
Cosine Similarity (similarity): Negatively correlated with dg, similar molecules should have embeddings with smaller Euclidean distances.
To quantify model performance, compute the Pearson absolute correlation coefficient between dy and these four representations. A higher absolute correlation indicates that the learned embedding space is more aligned with traditional molecular dissimilarity similarity representations, suggesting better representation learning.
Based the results presented in Table 13 model demonstrates competitive performance across the selected pre- pre-trained models, achieving the highest correlation among 3/4 representations, with values approaching 0.5, This indicates moderate correlation between the embedding Euclidean dis- tance and molecular dissimilarity and similarity representations. The absence a strong correlation suggests that embeddings capture broader range of molecular features beyond the structural similarities reflected in the selected representations.
Notably, MoL Mormer proves to be strong competitor, evidenced by its comparable performance. is important to highlight that their paper [Ross et al. [2022,, significantly higher correlation values, approaching 0.7, were reported. However, this can be attributed to their use of the same dataset,
34

---

Table 13: Experiemnt results of molecular similarity analysis, The Absolute Pearson correlation coefficient (between Euclidean distance and the four dissimllaiitySiimilarr representations) is reported as the metric. Values represent the mean and standard deviation over five runs.
MODEL
Tanimoto DISTANCE
MCS NUM AroMS
DICE SIMILARITY
COSINE SIMILARITY
MoL.ORRER
0.45 (o012)
(0012)
0.363
0.453
0.445
CHEMBERTA-2
(o.08
0.272
0.339
0.334
MOLBERT
0.43 (0,006)
0.275
0.437
0.414.0.006)
MuMo
(.490)
0.290
0.498
0.487 (0.06)
Layer
Layer
(a)
Token Index (Y -axxis)
Atlention
Layer 10
Layer 12
Layer
Layer 16
Layer
Layer
Layer
Layer
(b)
Token Index (Y -xaxis)
Layer 10
Layer
Layer
Layer 16
Token Index (X-axis)
Figure 9: Attention visualization of MuMo for two molecules. (a) D=CINCCCO)2-CC--C--2NCCC
and (b) O-CCCCCONCI--CCCCCCCCC
Each molecule is analyzed across all 16 layers, with attention scores plotted every two layers. The MuMo model starts interacting with GGFN information from layer
10M-PubChem Kim et al. 2016), for both pre-training and evaluation, which likely introduces dataset overlap and inflates the results.
For better comparison between MoLFormer and our model, we visualize results from both models in F The first row of the figure represents the visualization for our model, while the second row corresponds to MoLFormer. Each subplot shows the relationship between the Euclidean embedding distance and a molecular dissimilarity or similarity representation. For our model, the distributions appear more elongated and flattened along the diagonal, indicating stronger correlation. In contrast, the plots for Mol.Formes exhibit more rounded distributions, reflecting weaker correlations. This difference suggests that our model's 's embeddings are more effective, as evidenced by the stronger alignment with the molecular dissimilarity and similarity representations.
35

---

Table 14: Computing cost comparison in the pretraining stage. We compare model scale, dataset size, and estimated pretraining time across different molecular foundation models.
MODEL
SCALE
PRETRAIN DATASET SIZE
PRETRAIN TIME
MoLFoRMER--BL -BASE
44.28M
1.BB
16xV100-333 ESt. 104n
MoLFORMER-XX
86.75M
1.IB
16xV 100-300,,, 208a
UNIMOL
47.61M
19M
4xV100-322 100-32G, 20h
UNIMOL-V2
1.1B
884M
64xA 100-000 EST. 2 WEEKS
MuMo
505M
1.6M
4xA100-80G, 5H
Table 15: Computing efficiency comparison with Uni-Mol-v2. MuMo achieves comparable perfor- mance with substantially fewer parameters, less pretraining data, and drastically reduced compute time.
MODEL
MODEL Size
PRETRAIN DATA Size
COMPUTING RESOURCES
UNI-Mor-v2
I.IB 505M
884M FROM ZINC20
64XA100-8000 ESt. 1-3 WEEKS
MuMo
1.6M CHEMBL
4xA100-80CG, 5 HOURS
C.8 Attention Analysis for Fusion Effectiveness
To evaluate the effectiveness of stream fusion in our model, we visualized the self-attention scores across the 16 main layers of the MuMo model (comprising 8 Attention-Mamba layers and 8 multimodal fusion layers) when inferring dataset Kömezz-Bomarde two layers, providing a detailed view of how the model evolves through its hierarchical feature extraction process. This visualization enables us to track the progression of contextual relationships captured by the sequence flow along with its interaction with the structural fusion module using the Progressive Injection (PI) method.
molecules randomly selected from the 7INC-250K et al.) 20188. as shown in Figure Attention scores were plotted every
In the first eight layers, the model primarily focuses on self ceff-modeling within the sequence flow without incorporating topology or geometry information. During this stage, attention scores exhibit a relatively uniform distribution, indicating limited contextual understanding and lack of meaningful interatomic relationships. Starting from layer 9, the model begins interacting with the outputs of the structural fusion module, starting the hierarchical injection with other structural modalities. This interaction enables the model to capture more localized information, as evidenced by the increased attention weights near the diagonal in the fifth subfigure for both molecules in Figure Paa a) and b)). These highlights correspond to strong pairwise dependencies within local regions of the molecular structure. In subsequent layers, the sequence flow continues interact with topology and geometry flows while independently refining its representations. By the final layer, the model demonstrates the ability capture long-rat mangrrage contextual relationships, as seen in the attention scores for distant atom pairs. Additionally, the attention score of the [GI.]] token (the first token) becomes significantly stronger through the last layers, showing has captured rich representation in the molecular by the Progressive Injection (PI) method, which enhances the model's capacity for robust global representations. This progression underscores the model's s ability to integrate multimodal information effectively.
C.9 Training Cost Analysis
High efficiency in pretraining. As shown in Table 14 MuMo is highly efficient in the pretraining stage, requiring only 5 hours 1. 1.6M-ccale dataset using 4xA100 GPUs, significantly less than prior models that consume days or weeks but got better performance. Table [5] we compare the pretraining cost to the UniMol-v2 [Ji et al. 2024] model, showing our significant efficiency in pretraining stage.
High efficiency in finetuning. In the finetuning stage (Table 16), even on our largest benchmark dataset QM9 (133k samples), MuMo completes training in just 3.3 hours with modest GPU memory (14 GB per card). For smaller datasets (few thousand samples), training typically finishes in under 30 minutes. These results demonstrate that MuMo offers strong performance without prohibitive resource demands, making it practical for real-world use.
36

---

Table 16: Computing cost on QM9 in the finetuning stage. We report the hardware configuration, dataset statistics, and training runtime for fine-tuning MuMo (505M) on QM9.
ITEM
VALUE
DATASET
QM9
SAMPLE COUNT
133,855
CPU CORES
32
CPU
INTEL(R) XEON(R) PLATINUM 8480+
GPU COUNT
2
GPU TYPE
NVIDIA A A100-SXM4-80GB
MODEL PARAMETERS
50SM
TOTAL FLOPs
65,218,081 GF
EPOCHS
10
GLOBAL BATCH SIZE
64 14G PER GPU MIN
MAX GPU MEMORY USAGE
DATA PREPROCESS TIME
TRAINING TIME
3u 21M 51s
D Extended Experiments and Ablation Studies
To comprehensively evaluate the effectiveness of our proposed method and share our training expe- rience with other researchers, particularly the innovations in the structural fusion pipeline and PI (Progressive Injection), we conducted an extensive series of ablation studies. These experiments not only validate the contributions of our approach but also provide valuable insights into training strategies and model development. Our analysis systematically investigates the impact of various components and innovations, including pretraining strategies, multimodal interaction mechanisms, sequence data types, pretraining datasets, model capacity, and some hyperparameter configurations.
To ensure the rigor and scientific validity of our experiments, we began by focusing on multimodal fusion. Specifically, starting from the best-performing model, we progressively removed one modality time to establish three baseline models: the complete model (our proposed method), model fusing sequences and 2D graphs. and a sequence-only model. Based on comparisons of these three baselines, we further conducted ablation studies by reducing or modifying individual modules within each model to highlight the contributions of different components to the final performance. In this section, we primarily use five medium-sized datasets from MoleculeNet, applying the recommended split methods: scaffold split for BACE and BBBP, and random split for the others.
D.1 Contribution of Multimodal Combinations
In this section, we demonstrate the effectiveness of the proposed MuMo model and use the best model as baseline for ablation experiments. The full MuMo model represents our unaltered approach. To investigate the impact of different modalities, we conducted the following variations: (1) removing geometric information and the fusion of geometric and 2D topological flow, and (2) utilizing only SMILES sequences for modeling without using any fusion methods. These two models are labeled as Ablation Baseline I and 2 (AB1 and AB2) for other ablation experiments, with MuMo itself designated as Ablation Baseline 3 (AB3). All other conditions were kept identical. After the same pretraining process, the models were evaluated on five datasets, comprising three classification tasks and two regression tasks from MoleculeNet, to ensure a comprehensive assessment.
From the results presented in Table 17 it is evident that the model without geometric information experiences varying degrees of performance degradation across datasets, with a particularly significant drop observed on the BACE dataset. This indicates that geometric information plays a critical role in capturing subtle molecular properties for certain classification tasks. Comparing the SMILES-onf model with the sequence+2D graph fusion model reveals minimal differences in performance on classification tasks but a substantial decline in regression tasks. This suggests that accurate numerical predictions in regression heavily depend on the model's 's ability to capture the full 2D topological structure of molecules. These findings underscore the indispensable roles of both 2D topology and 3D modeling in molecular representation learning.
37

---

Table 17: Ablation results on multimodal combinations. BACE, BBBP, and Clintox are classification tasks (AUROC, higher is better), while ESOL and LIPO are regression tasks (RMSE, lower is better). The three methods include ABI: SMILES Only beqqeencebbssedee AB2: SMILES Graph (sequence and topology), and MuMo (sequence, topology. geometry).
CLASSIFICATION
REGRESSION
METHOD
BACE
BBBP
CLINTOX
ESOL
LIPO
ABI SMILES ONLY
0.798...115)
0.931 (0000)
0.958
1.793
0.844
AB2 SMILES + GRAPH
o..22)
0. 00946)).008)
go05)
0.597
0.596
AB3 MuMo
0.849
0.957
0. 0.985
0.536
0.577
Table 18: Ablation results of pretraining contribution. Results are presented with AUROC for classification tasks (BACE, BBBP, CLINTOX) and RMSE for regression tasks (ESOL, LIPO). Standard deviations are reported over three runs.
CLASSIFICATION
REGRESSION
PRETRAINING
FINETUNING
BACE
BBBP
CLINTOX
ESOL
LIPO
8.89 (0.014)
0.957
1.985 (0011)
0.536
(o.07)
X
0.807
0.930
0.928
2.568
0.887
D.2 Pretraining Ablation Studies
Recent works, such MolBERT Fabian al. 2020) and MoLFormer (Ross al. 2022]. have shown that the effectiveness of large-scale molecular pretraining relies heavily on massive datasets, often exceeding 1 billion molecules. In contrast, our proposed method, MuMo, achieves superior performance on downstream tasks with significantly less pretraining data. By leveraging only 1.6M molecules from the ChEMBL dataset [Gaulton et al. 2012 MuMo demonstrates that pretraining on a much smaller scale can be both efficient and highly effective. Furthermore, ablation studies on our pretraining approach validate the robustness and efficiency of MuMo, highlighting its ability to extract meaningful representations even with limited data.
D.2.1 Pretraining Effectiveness
To evaluate the necessity pretraining, we compared two models: one fully pretrained on the 1.6M ChEMBL dataset and the other directly fine-tuned on downstream tasks without any pretraining (zero-shot fine-tuning). This comparison isolates the impact of pretraining by examining influence downstream performance and training dynamics. Both models share identical architectures, hyperparameters, and training configurations during fine-tuning to ensure fair and controlled evaluation.
The results in Table 18 demonstrate the critical role of pretraining enhancing molecular property prediction, with particularly significant effects on regression tasks. For classification tasks such as BACE and BBBP, pretraining yields consistent but modest improvements, increasing AUROC scores by 5.9% and 3.5%, respectively. These gains suggest that pretraining effectively captures foundational molecular patterns and functional group features, which improve generalization. However, impact is most pronounced in regression tasks like ESOL and LIPO, where RMSE reductions from 2.568 to 0.536 and 0.887 to 0.577. This reflects the importance pretraining in encoding high-resolution spatial and geometric molecular features, essential for accurately modeling continuous properties such as solubility and lipophilicity.
The differential impact of pretraining stems from the varying nature of the tasks. Classification tasks rely on discrete structural patterns, which fine-tuning alone can partially capture, whereas regression tasks demand precise geometric and spatial modeling, making pretraining indispensable. By leveraging the Unified Graph and molecular segmentation, pretraining enables the model to encode both local interactions and global molecular topology effectively. These findings underscore the necessity of pretraining for complex molecular representations, particularly in tasks requiring high-resolution structural information.
38

---

Table 19: Ablation results on pretrain datasets. Downstream task performance comparison of models using different pretraining datasets. The metrics for BACE, BBBP, and Clintox are AUROC (higher better), while ESOL and LIPO use RMSE (lower is better). The models include SMILES- -nnly (AB1, sequence-basee and MuMo (sequence, topology, and geometry).
CLASSIFICATION
REGRESSION
SIZE OF PRETRAIN DATASETS
BACE
BBBP
CLINTOX
ESOL
LIPO
10M PUBCHEM SEQUENCE ONLY
(.849
0.856g.037))
0.677
2.543
0.797
1.6 CREMBL- - SEQUENCE ONLY
0.798
0,958 g.931..005)
(.793
0.844..033)
MIXDATASETS MuMo
0,853
0.956g.038)
0.972.o.110
(.607
0.604
10M PUBCHEM- MuMo
0.784
0.946
(.979(o.134)
0.635
0.611
1.6M CHEMBL. MuMo
0.849 (0.044)
0.957 (0.011)
0.985
0.5.6
0.577
D.2.2 Scale and Source of Pretrain Datasets
To evaluate the influence of the scale and type of pretraining datasets on downstream molecular property prediction tasks, we conducted experiments using models pretrained on datasets of varying sizes and sources. Specifically, we compared two model architectures: SMILES MMlLEs-Only (AB1), which relies solely on sequence representations, and MuMo, which is our best model. Pretraining datasets included PubChem (10M molecules) (Kim et al. 2016), ChEMBL (1.6M molecules) [Gaulton et al. 2012), and a curated "MixDatasets", which integrates these three datasets. We assessed model performance on five benchmark datasets (BACE, BBBP, Clintox, ESOL, and LIPO), covering both classification and regression tasks. This experiment is necessary to investigate how pretraining data characteristics- robustness of models in molecular property prediction, which is critical for ad advancing drug discovery and materials science applications.
as scale, diversity, and feature richness-affe
the generalization and
The results in Table 19 highlight the significant impact of pretraining dataset scale and type on downstream task performance. Models relying solely on SMILES sequences perform reasonably well on certain tasks, such as the AB1 (sequence only) model trained on the 10M PubChem dataset achieving an AUROC of 0.849 on BACE. However, their performance on other tasks is relatively limited. This suggests that sequence-only representations may lack the expressiveness needed for diverse property predictions. contrast, the introduction of the multimodal fusion strategy in the MuMo models substantially improves performance across tasks. For instance, the MuMo model trained on the 1.6M ChEMBL dataset achieves leading results on Clintox and ESOL, indicating superior capability in toxicity and solubility predictions.
Interestingly, while the performance differences between models pre-trained on different datasets are not particularly large, the 1.6M ChEMBL dataset consistently outperforms others, especially when used with the MuMo model. This can be attributed to ChEMBL's curated nature, as it specifically focuses on bioactive molecules with experimentally validated activity against biological targets. This high-quality, domain-relevant data likely provides more meaningful molecular patterns and task-specific signals for the model to learn. In contrast, larger datasets like the 10M PubChem, which contain more diverse and potentially noisy molecular structures, may introduce heterogeneity that dilutes the relevance of features for downstream tasks. The ChEMBL dataset, combined with multimodal representation learning approach, allows the model to effectively generalize and achieve strong performance across various tasks. This highlights the importance of data quality and relevance over sheer size in molecular pretraining.
D.2.3 Pretraining Strategy
This experiment explores the impact of different pretraining tasks and pooling methods on the performance of the sequence-based ablation baseline model (AB1). Two pretraining tasks are considered: Masked Language Modeling (MLM), which involves masking a portion of the input sequence and training the model to predict the masked tokens based on the context, and Next Token Prediction (NTP), where the model predicts the next token in the sequence to learn sequential dependencies. For MLM, two pooling methods are tested during the fine-tuning phase: mean pooling, which averages all token embeddings in the sequence to form the final representation, and GTK token pooling, which uses the embedding of the first special token (GTK) as the final representation. The
39

---

MLM Mean
(6)
Epoches
1.75
MLA CLS
Epoches
1.25
1.00
0.75
0.2
0.00
E50
040
BACE
D98P
Cliox
ESOL
UPO
(4)
(4)
BSM 18 Layers)
370M (15 Layers)
730M Layers)
1.0
1.5
0.5
1.4
0.5
0.4
0.2
0.0 RACE B38P
Clitto
EGCL LIRO
Figure 10: Ablation results on pretraining Epochs, pooling methods, model size, and pretraining strategies. (a) The impact of pretraining epochs model performance. (b) The effect of pooling methods during fine-tuning. GTK (first token). SEP (last token), mean pooling, max pooling, and combined methods (GTK + Mean + Max SEP) are compared. (c) Influence of model size. Compared to the larger model using 32 layers and the smaller model using layers. (d) The effect of pretraining strategies Masked Language Modeling (MLM) and Next Token Prediction (NTP) with pooling methods. Metries: AUROC (higher is better) for classification datasets (BACE, BBBP, Clintox) and RMSE (lower is better) for regression datasets (ESOL, LIPO).
performance these variations is evaluated across five datasets, with classification tasks (BBBP, Clintox, and ESOL) using AUROC and regression tasks (LIPO) using RMSE. This setup aims to assess how pretraining tasks and pooling strategies influence downstream performance.
The results for Figure 10 (a) show that the NTP pretraining task generally underperforms compared to MLM, achieving the lowest overall performance across the evaluated datasets. This can be attributed to the nature of the task itself. NTP focuses predicting the next token in the sequence, which emphasizes sequential dependencies but lacks the capacity capture broader structural information about the molecule. In contrast, MLM allows the model to learn richer contextual relationships by considering both preceding and following tokens, which crucial for representing the complex structural and chemical properties of molecules. By masking random tokens, MLM forces the model integrate information across the entire sequence, enabling a more holistic understanding of molecular features.
Among the pooling methods, GTK token pooling consistently outperforms mean pooling. This difference can be explained by the way each method aggregates information. GTK token pooling uses dedicated GTK token trained explicitly to encode global representation the input sequence, allowing it to serve as a focused summary of the molecule's overall properties. In contrast, mean pooling averages the embeddings of all tokens, which can dilute critical information and make it harder for the model to distinguish important structural or functional features. The superior performance GTK pooling suggests that the specialized GTK representation better suited for tasks requiring a compact yet informative molecular encoding.
D.2.4 Pretraining Epoches
Results from Figure TO b)) show that the difference between and 5 epochs pretraining is negligible across all tasks, suggesting that the model converges quickly. However, extending pretraining to 10
40

---

epochs negatively impacts regression tasks, particularly ESOL, where performance drops significantly. This trend aligns with the loss curves from Figure 4 which demonstrate that the model converges rapidly and maintains stable loss values with minimal fluctuation. Prolonged pretraining provides diminishing returns and can even degrade performance on certain tasks, likely due to overfitting or excessive adaptation to the pretraining data. This behavior highlights key advantage of our model: its ability to achieve strong performance with minimal pretraining epochs, making it computationally efficient and robust.
D.3 Impact of Model Size
Results from Figure T0 c) demonstrate that model size has a significant but nuanced impact on per- formance. Smaller models, such as the 85M parameter model with 8 layers, generally underperform due to limited capacity capture the complexity of molecular representations. However, excessively large models, like the 790M parameter model with 32 layers. also fail to consistently outperform the 370M parameter model with 16 layers, particularly on tasks like Clintox and LIPO. This suggests that overly large models may suffer from overfitting or diminishing returns when scaling parameters. The 370M model strikes balance between capacity and generalization, achieving competitive performance across both classification (e.g., BBBP) and regression (e.g., ESOL) tasks. This indicates that there is an optimal model size for capturing molecular structure and properties without incurring computational inefficiency or overfitting risks.
D.4 Impact of Pooler Methods in Finetuning
This experiment shown in Figure 10 d) evaluates the impact of different pooling strategies during the fine-tuning phase on downstream task performance. For regression tasks the addition of the SEP token increases the values, indicating negative effect on regression accuracy. Similarly, the inclusion of mean and max pooling does not outperform the simpler GTK- -olly (global token) approach, as the performance remains worse when combining multiple pooling strategies. This outcome underscores the importance of the GTK token for providing compact and enriched global representation, which is particularly critical for regression tasks. Regression tasks are more sensitive the accuracy global representations and demand higher precision in capturing geometric and topological information. The MuMo model's 's multimodal fusion process explicitly strengthens the GTK token through injected information during pretraining, making the GTK token crucial for achieving accurate global molecular property predictions regression tasks.
For classification tasks, the performance differences across pooling methods are relatively minor. This because the Injection-Enhanne Enhanced mechanism during pretraining plays a dominant role in strengthening the GTK token with rich multimodal information, surpassing the contribution of the symmetric cross-altention interaction. As result, the GTK token alone provides a sufficiently strong and compact representation for coarse-grained classification tasks. The addition of other pooling methods, such as SEP, mean, and max pooling, adds minimal benefit, as the GTK token already captures the essential sequence-level features needed for classification.
D.5 Impact of Sequence Data Type
This experiment evaluates the performance of sequence-only model (AB1) using three different molecular sequence representations: SMILES, SELFISH, and Morgan fingerprint while keeping all other conditions identical. The goal is to understand how different sequence data types influence the performance of molecular representation learning.
SMILES is widely used linear notation that encodes molecular structures text strings, capturing atom connectivity and bond information. SELFISH is a sequence representation derived from SMILES, optimized for specific tasks by restructuring the sequence to enhance information extraction. Morgan fingerprint, in contrast, is a fixed-length vector representation generated from molecular graphs, capturing substructural patterns but lacking the sequential structure of SMILES.
Table 20 shows that SMILES achieves the best overall performance, particularly excelling in classifi- cation tasks. While SMILES demonstrates some limitations in regression tasks, such as ESOL, its performance remains competitive, showeasing its versatility as a molecular representation. SELFISH also performs well but slightly trails SMILES, particularly in classification tasks, while Morgan
41

---

Table 20: Ablation results on different sequence data types. Performance comparison of models using SMILES, SELFISH, and Morgan fingerprint. The metrics for dataset BBBP and Clintox are AUROC (higher is better), while ESOL and LIPO use RMSE (lower is better).
CLASSIFICATION
REGRESSION
SEQUENCE DATA TYPE
BBBP
CLINTOX
ESOL
LIPO
SMILES (AB1)
0.931 (0,00))
0.958
(.793
0. (.84400003
SELFISH MORGAN
0.914
0.910
(..93)
0.84 (.804
0.903
(o.077o.135
0.939
0.880
Table 21: Ablation results on data type combinations for multimodal modeling. Full MuMo is our best model, which combines SMILES, 2D graph, and 3D Geometry information. The metries for BACE, BBBP, and Clintox are AUROC (higher is better), while ESOL and LIPO use RMSE (lower better).
CLASSIFICATION
REGRESSION
MULTIMODAL CHOICE
BACE
BBBP
CLINTOX
ESOL
LIPO
SMILES-GRAPH
0.780
0,946,0008)
0.97 (0015))
0.597
0.596
SMILES FINEEERRIIIR GRAPH
0.745
0.919 (0000)
0.969
0.646
(.633
FINGERPRINT GRAPH+ GEOMETRY
0.824
0.908
0.996
g.030
(au44)
FULL MuMo
0.849.0.014)
(.957(0011)
0,985
0.536
0.577
fingerprint struggles in classification tasks and shows inconsistent results in regression tasks. This suggests that the sequential information in SMILES provides more robust foundation for general molecular property prediction across both task types.
Results from Table 21 demonstrate that incorporating Morgan fingerprint into the multimodal combi- nations does not significantly improve performance and, in some cases, leads to a slight degradation. For instance, the combination of SMILES Fingerprint Graph underperforms compared SMILES + Graph alone, particularly on regression tasks. This suggests that the Morgan fingerprint, being a fixed-length vector representation, may not effectively complement other modalities like 2D topology graphs and 3D geometry, which provide richer structural and spatial information. The best results are achieved by the MuMo model, which combines SMILES, 2D graph, and 3D geometry, highlighting the importance of these complementary modalities in capturing molecular features comprehensively.
D.6 Impact of Key Modules
In this section, we present five modules that have been shown to enhance the representational effectiveness of the model during training. demonstrate the impact these modules and share our insights with peers, conducted series of experiments. Under identical conditions, modified only one module of the model at time and evaluated its performance on the corresponding datasets.
Table 22: Ablation results of three small modules. The effect of the substructure-level tokenizer, bi-attention, and different graph processing modules (MPNN and GCN). The metrics for BACE, BBBP, and Clintox are AUROC (higher is better), while ESOL and LIPO use RMSE (lower is better).
MODULE
DETAIL:
AvG. DROP CLASSIFICATION
AvG. DROP REGRESSION
SELF- ALTENTIOON
0.00%
0.00%
ATTENTION
BI-ATTENTION
0.88%
-18.68%
MPNN
0.00%
0.00%
GRAPH PROCESS
GCN
-408%
-11.37%
LEV
0.00%
0.00%
TOKENIZER
CHARACTER-LEVEL
-4.28%
-9.05%
42

---

(a)
(b)
BACE
BACE
ESCL
ESOL
LPO
Mc Mo-Mamba
UPO
A52
wD Mi Segrnentzion
Fond Gragh Encoder
Values
Figure II: Impact of (a) molecular segmentation and (b) graph learning strategy. The metrics for BACE, BBBP, and Clintox are AUROC (higher is better), while ESOL and LIPO use RMSE (lower better).
Substructure Partitioning. In this study, we proposed the Geometry Substructure Partitioning method to capture molecular features at the geometric substructure level. To evaluate its importance, we conducted an ablation study where the MuMo model served as the baseline, and the experimental group removed the fusion of 2D graph and geometry substructure modeling, retaining only the inter- action their global information with sequence information. The results illustrated in Figure aa) showed performance drops across all five datasets, highlighting the critical role of this module, particularly for regression tasks that demand precise and fine-grained modeling.
Graph Learning Strategy. This strategy enhances the model's ability dynamically learn graph representations across layers by updating the Unified node features after the symmetric Cross- Attention fusion at each layer, rather than relying static processing (fixed encoder). The updated node features are further refined in subsequent layers, allowing the model to better capture intricate and continuous relationships within molecular graphs. As shown in Figure 11 TT)), this strategy leads to performance gains across all datasets, particularly in regression tasks, by leveraging the progressively enriched node features for more precise and detailed molecular representation.
Bi-Attention. This method extends standard self-attention by introducing bidirectional mechanism, where sequence features are processed with normal self-attention and then passed through reversed sequence self-attention layer, followed by fusion of the two outputs. While this approach adds cross- directional context, our experiments show that it slightly improves classification tasks but negatively impacts regression tasks, which is illustrated in Table 22 This suggests that the added bidirectional processing may introduce noise unnecessary complexity, particularly for tasks requiring precise global property predictions. Consequently, this method was not adopted in our model.
Graph Processing Module (MPNN vs. GCN). In the Unified module, we evaluated the choice of Message Passing Neural Networks (MPNN) versus Graph Convolutional Networks (GCN) for message passing. MPNN consistently demonstrated superior performance across all tasks due to its ability to effectively capture intricate node relationships and propagate richer information through the graph. In contrast, GCN introduced significant performance degradations, which are shown in Table 22 particularly in regression tasks, where precise modeling of molecular structures is critical. This highlights the importance of using MPNN for robust and detailed feature extraction in the Unified module.
Substructure-Levéè Tokenizer. The substructure-levè tokenizer for the SMILES sequence outper- forms the character-level tokenizer, which struggles with both classification and regression tasks, as Table shows. Unlike the character-level approach, which splits SMILES into individual char- acters and loses critical structural context, the substructure-level tokenizer treats special ions and atoms as individual tokens, preserving their unique chemical and structural properties. This results in significantly better performance by capturing meaningful substructure information essential for downstream tasks.
43

---

Table 23: Impact of the ratio of hierarchical fusion layers. The metrics for BACE, BBBP, and Clintox are AUROC (higher is better), while ESOL and LIPO use RMSE (lower is better).
CLASSIFICATION
REGRESSION
LAVER NUM FUSIoN: SEQUENCE
BACE
BBBP
CLINTOX
ESOL
LIPO
0.75
0.8 810.02))
(0954))006)
0.992m.010)
0.587
0.588
0.25
0.841 0.841,00003)
0.961 0.961 (0.008)
0.994
0.617.0.016
(.602
0.5 (MuMo)
0. 0.849 (0004))
0.957 (0.11)
0.985 (0.01))
0.536
0.577
Table 24:
Overlapping evaluation on MoleculeNet datasets (classification). "Overlap Rate" denotes the proportion of molecules overlapping with the pretraining corpus. Results are AUROC (1) with standard errors in gray subscript.
MODEL: MoMo
BACE
BBBP 7
CLINTOX
SIdER 7
Tox21 1
OVERLAP RATE
0.00%
4.61%
1,42%
1.60%
67.54%
ORIGINAL PERFORMANCE
0.849m014
0.957omm1
0.985umm1
0.677a009
0.834a.09
CLEAN PERFORMANCE
0.849b014
0.9600.099
0.996v.011
0.675o011
0.8396.009
PERFORMANCE CHANGE
0.00%
+0.31%
+1.12%
-0.30%
+0.60%
Table 25:
Overlapping evaluation on MoleculeNet datasets (regression). "Overlap Rate" denotes the proportion of molecules overlapping with the pretraining corpus. Results are RMSE (L) with standard errors in gray subscript.
MODEL: MuMo
ESOL J
FREESOLV 1
LIPO
QM7 J
QM8 J
QM9 J.
OVERLAP RATE
37.67%
29.28%
6.90%
4.61%
2.09%
0.58%
ORIGINAL PERFORMANCE
0.536o
1.082m.os
0.448
42.800.
0.00110.000
0.0030n.0000
CLEAN PERFORMANCE
0.342oo81
0.796n.056
0.439b.006
45.670.5
0.01120.000
0.00330.0000
PERFORMANCE CHANGE
+36.2%
+26.4%
+2.01%
+6.70%
+0.90 %
0.00%
Table 26: Ablation on Backbone Generalizability. Results on BBB (AUROC T), PGP (AUROC T). Caco2 (RMSE L), and PPBR (RMSE L). Standard errors are shown gray subscript.
MODEL
BBB 1
PGP
CAco2 J
PPBR
AvG. DROP
MuMo (ArTN_ MAMBA, OURS)
0.8
0.9426.019
0.3
7.324o.31
0.00%
MuMo (TRaNSFORMeR)
0.8670.01
0.899%.04
0.3176.00
8,489h.591
6.66%
MuMo (VANILLA MAMBA)
0.869
0.9088.020
0.3960045
7.465u.33
8.65%
MAMBA-ONI (w/o FUSION)
0.8 13.19
0.8760.02s
0.832
11.54a72
59.55%
MuMo (TRANSPORMER, w/o FUSION)
0.843
0.889
0.644
9.892am01
37.84%
D.7 Impact of Number of Fusion Layers
This experiment evaluates the impact of the number of fusion layers in the model on its performance across various molecular property prediction tasks. The results from Table indicate that the optimal number of fusion layers is crucial for achieving the best performance, neither too few nor too many layers consistently yield superior results. The model demonstrates clear peak in performance with a balanced fusion layer configuration, highlighting the importance of effective feature integration for multimodal molecular representations.
D.8 Backbone Generalizability
Ablation on the backbone. As shown in Table MuMo with Transformer and Vanilla Mamba backbones both achieve competitive results, confirming that our fusion design indeed model- agnostic.
Ablation on fusion. In contrast, when the fusion module removed, both Mamba-only and Transformer-only baselines suffer large performance drops, which are up to 59.55% and 37.84%,
44

---

Table 27: Ablation on asymmetrical vs. symmetrical fusion approaches. Results are reported on BBB (AUROC T), PGP (AUROC 1), Caco2 (RMSE L), and PPBR (RMSE L). Standard deviations are shown in gray subscript. The best result is highlighted.
MODEL
TYPE
BBB 1
PGP 1
Caco2 J
PPBR L
AvG. DROP
MuMo
ASYMMETRIC
0.899
0,9420019
0,315
7.3240.325
0.00%
MuMo-FG
ASYMMETRIC
0.878
0.901eom
0.412ao51
7.935u.297
11.96%
MyMo-CE
SYMMETRIC
0.889
0.9170016
0,475
8.2730.112
16.88%
MrMo-CB
SYMMETRIC
0.849
0.891oo2s
0.601
9.13203390
31.60%
Table 28:
Evaluation ESOL and BBBP datasets using different conformer optimization tools. Results are reported with standard errors in gray subscript. Std. Dev denotes the standard deviation across conformer settings, and Max A is the largest difference observed.
TASK
MODEL
MMFF94
UFF
No-OPTıMIZE
Sı.. DEV 1
MAX L
ESOL (1)
UniMot
0.7696.153
0.790o.113
0.939b.191
0.0927
0.170
ESOL (L)
MuMo
0.5360
0.5500o02
0.585
0.0252
0.049
BBBP (1)
UsIMor
0.889
0.831
0.733
0.0789
0.156
BBBP (1)
MuMo
0.962
0.952x.09
0.941 0.000
0.0105
0.021
Table 29: Robustness to 3D conformer noise. Performance under increasing levels of Gaussian noise (in À) added to atom coordinates. Results are reported with standard deviation in gray subscript.
MODEL DATASET
0À
0.05A
0.1À
0.2À
STD. DEV
MAX &
UNIMoL ESOL (L)
0.769e.133
0.765o.
0.760o.44
0,831e2200
0,0334
0,071
MuMo ESOL (L)
0.536mmm1
0.530n.00
0.540
0.544
0.006
0.014
UNIMoL BBBP (1)
0.889b.025
0.878n.022
0.890
0.787a.c9
0.0496
0.103
MuMo BBBP (T)
0.962
0.960m.007
0.961
0.954
0.0036
0.008
This highlights that the key performance gains come from our structured fusion strategy, not from the choice of backbone alone.
respectively (Table 26
Overall, while our hybrid Attn-Mamba design yields the strongest results, our ablations demon- strate that the proposed fusion scheme can generalize across architectures, validating its standalone effectiveness.
D.9 Impact of Asymmetric Fusion
To investigate the role of fusion strategy, we compare our asymmetric injection-enhanced design against symmetric concatenation-basel alternatives. As shown in Table 27 the full MuMo with asymmetric fusion consistently achieves the best performance across all benchmarks. By contrast, symmetric strategies that concatenate modalities either at the beginning (MuMo-CB) or the end (MuMo-CE) lead to large performance drops, especially on regression tasks such as Caco2 and PPBR. The fixed-graph variant (MuMo-FG) also underperforms due to its inability to jointly adapt representations. These results confirm that asymmetric injection not only preserves codaality-speciffi information but also enables effective cross- cross-modal enrichment, leading superior generaliz- ability.
D.10 Potential Data Overlap Analysis
Experiment Results. We analyzed all MoleculeNet datasets for potential overlap with our ChEMBL-I.6N pretraining corpus, and re-ran experiments after removing overlapping molecules. Surprisingly, the average performance increased by 6.81% % across datasets rather than dropping (Tables 24 and 25
This confirms that our pretraining task (masked language modeling) does not leak label information even under worst-case overlap scenarios (c.g.. T,x21, where the overlap reached
Explanation.
45

---

67.5%). Since no downstream supervision is involved during pretraining, the model cannot memorize labels from ChEMBL. Moreover, given the scale of ChEMBL and our relatively short training schedule (2 epochs), it is unlikely that the model memorized specific molecular structures.
Observation. Notably, the largest gains appeared in datasets with the highest overlap, such as ESOL (+36.2%) and Freesolv (+26.4%). We hypothesize that overlapping molecules may have been harder examples, and their removal led to cleaner evaluation splits. This highlights both the robustness of MuMo and the soundness of our pretraining corpus design.
D.11 Ablation on Conformer Settings and Robustness to Geometric Perturbations
Effect of Conformer Generation Tools. We evaluated MuMo and UniMol under three common conformer settings: (1) MMFF94-minimized, (2) UFF-minimized, and (3) ETKDG without optimiza- tion. As shown in Table 28 MuMo maintains stable performance across all conformer sources and exhibits notably smaller fluctuations compared to UniMol. For example, on ESOL, MuMo's 's standard deviation across conformer sources is only 0.0252, much lower than UniMol's 0.0927. This indicates that our injection-enhaneed design provides robustness against geometric inconsistencies, ensuring reliable predictions even when different conformer generation pipelines are used.
Impact of Conformer Noise. We further test sensitivity to conformer noise by perturbing atom positions with Gaussian noise varying magnitudes. As summarized Table 29 UniMol suffers significant degradation as noise increases (e.g., ESOL from 0.769 to 0.831 RMSE). while MuMo's performance remains nearly unchanged (Std. Dev = 0.006, Max 0.014). Similar trends are observed on BBBP. These results confirm that MuMo is more robust to 3D geometry noise, making better suited for real-world scenarios where generated conformers may be approximate.
46


Reviews:
[{"review_id": "aT56vV09BP", "reviewer": "Reviewer_A1Jg", "summary": "The paper introduces MuMo, a multimodal molecular representation model that fuses SMILES, 2-D topology and 3-D geometry. A Structured Fusion Pipeline (SFP) first builds a “unified graph” that merges 2-D bonds and 3-D length/angle triplets, plus a geometry-aware BRICS-style sub-graph partition. A Progressive Injection (PI) mechanism injects the structural prior into a Mamba-based sequence backbone only after early layers have encoded SMILES semantics, aiming to avoid “modality collapse.” Experiments on 21 TDC / MoleculeNet tasks show an average +2.7 % over prior baselines and strong gains on conformer-sensitive LD50.", "questions": "1. Is there any comparison against a Transformer encoder with the same PI + SFP? It would be interesting to tell whether Mamba itself, rather than the fusion scheme, drives the reported gains.\n2. The authors claim robustness, but only RDKit MMFF conformers are evaluated. It would be better if they evaluate on QM-optimized geometries, noisy random torsions, or single conformer per molecule.", "limitations": "yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n1. The expriments are run on extensive benchmarks, the results are promising\n2. The methods sound rational and theoretical insights are provided\n3. The two challenges unreliable conformers and modality imbalance are important and interesting\n4. The paper is well-written with clear motivation\n\nWeakness:\n1. There is no comparison to recent SE(3) GNN + sequence hybrids (e.g., Uni-Mol-v2)\n2. The novelty of the method is limited, it's mostly based on given backbone. And the contribution mostly comes from the fusion module.\n3. Although the benchmark is large, all of them are physicochemical/ADMET. Catalytic activity, reaction yield, or crystal property datasets are absent, so generality remains speculative.\n4. ChEMBL-1.6 M overlaps with MoleculeNet/TDC targets, so there might be some information leakage.", "was_revised": true, "final_justification": "The problems are mostly addressed with new experiment results.", "rebuttal": "We thank the reviewer for recognizing that this is a well-motivated and clearly written study, addressing two key challenges, unreliable conformers and modality imbalance.\n\nTo save space, standard deviations are omitted below.\n\n**[W1]** Comparation with Baseline Uni-Mol-v2\n\nWe evaluated our model on the official data splits of QM9 from Uni-Mol-v2 , using their published data splits and metrics. In Table 1, MuMo achieves **superior performance on 6/8 tasks.**\n\n> Table 1. Evaluation on QM9 dataset with Uni-Mol-v2 and other baselines.\n\n| Model         | HOMO/LUMO/GAP (↓) | alpha (↓) | Cᵥ (↓)    | mu (↓)    | R² (↓)    | ZPVE (↓)   |\n| ------------- | ----------------- | --------- | --------- | --------- | --------- | ---------- |\n| Uni-Mol       | 0.0043            | 0.363     | 0.183     | 0.155     | 4.805     | 0.0011     |\n| Uni-Mol2 570M | 0.0036            | 0.315     | 0.147     | 0.089     | 4.523     | 0.0005     |\n| Uni-Mol2 1.1B | 0.0035            | 0.305     | 0.144     | **0.089** | **4.265** | 0.0005     |\n| **MuMo 505M** | **0.0030**        | **0.283** | **0.126** | 0.391     | 18.08     | **0.0005** |\n\nTable 2 shows MuMo achieves better performance than Uni-Mol-v2 with **half the parameters** and **1/400 the pretraining data**, showing its efficiency and strong multimodal fusion capability.\n\n> Table 2. Computing efficiency comparision with Uni-Mol-v2.\n\n| Model      | Model Size | Pretrain Data Size   | Computing Resources                         |\n| ---------- | ---------- | -------------------- | ------------------------------------------- |\n| Uni-Mol-v2 | 1.1B       | 884M from ZINC20     | 64x A100-80G, Est. 1~3 weeks from its paper |\n| **MuMo**   | **505M**   | **1.6M from ChemBL** | **4x A100-80G, 5 hours**                    |\n\nAll these results demonstrate that MuMo offers a competitive and efficient alternative to Uni-Mol2, validating its strength.\n\n---\n\n**[W2]** The novelty mostly stemming from the fusion module. \n\nWe thank the reviewer for this thoughtful concern. As our work specifically targets **multimodal molecular representation**, it is natural that one of the contributions lie in the fusion mechanism. Beyond this, we also propose a **modified MPNN** (Message Passing Neural Network) to effectively integrate both 2D and 3D geometry, capturing global and local structural information. And a **hybrid modeling block** that combines **self-attention and cross-attention** for better structural injection. Each module is validated in detailed ablation studies (In paper Sec. 4.3, Appendix D). \n\n---\n\n**[W3]** Broader Chemical Benchmarks: Catalytic activity, reaction yield, or crystal property.\n\nWe extended MuMo to accept reaction-level inputs and evaluated it on four datasets from **Reaxtica**[1]. These datasets cover **catalytic activity**, **reaction yield**. MuMo achieves **the best on all the tasks** (Table 3), demonstrating its strong transferability.\n\n**Crystal property** datasets are mostly **inorganic and do not support SMILES or RDKit-based analysis**, so we leave them for future extension.\n\n> Table 3. Evaluation on catalytic activity and reaction yield task.\n\n| Models   | BHC - R² (↑)      | Models        | CPA - MAE (↓)      | Models   | HTE - R² (↑)      |\n| -------- | ----------------- | ------------- | ------------------ | -------- | ----------------- |\n|          | Reaction Yield    |               | Catalytic Activity |          | Reaction Yield    |\n| Reaxtica | 0.94              | Reaxtica      | 0.144              | Reaxtica | 0.87              |\n| MFF      | 0.92              | MFF           | 0.144              | rxnfp    | 0.81              |\n| rxnfp    | 0.95              | Denmark et al | 0.152              | DRFP     | 0.85              |\n| **MuMo** | **0.952 (0.002)** | **MuMo**      | **0.144 (0.000)**  | **MuMo** | **0.873 (0.002)** |\n\n**These results suggest that our multimodal design generalizes well even outside its original single-molecule scope.**\n\n[1] Lin K, Li J, Lin H, Pei J, Lai L. Reaxtica: A knowledge-guided machine learning platform for fast and accurate reaction selectivity and yield prediction. ChemRxiv. 2022. \n\n---\n\n**[W4]** Potiential overlap between pretraining and funetuning datasets.\n\nWe analyzed all MoleculeNet datasets for overlap with our ChEMBL-1.6M pretraining set, and re-ran experiments after removing overlapping. \n\nSurprisingly, **the average performance increased by 6.81%** across datasets (Table 4&5). This confirms that our pretraining task (MLM) **does not leak label information**. \n\nMoreover, ChEMBL is large and we only trained for 2 epochs, making it unlikely for the model to memorize specific molecules. \n\n> Table 4. Overlapping evaluation on MoleculeNet datasets - classification.\n\n| Model: MuMo          | BACE (↑) | BBBP (↑)   | CLINTOX (↑) | Sider (↑) | Tox_21 (↑) |\n| -------------------- | -------- | ---------- | ----------- | --------- | ---------- |\n| Overlap Rate         | 0.00%    | 4.61%      | 1.42%       | 1.60%     | 67.54%     |\n| Original Performance | 0.849    | 0.957      | 0.985       | 0.677     | 0.834      |\n| Clean Performance    | 0.849    | 0.960      | 0.996       | 0.675     | 0.839      |\n| Performance Change   | 0.00%    | **+0.31%** | **+1.12%**  | -0.30%    | **+0.60%** |\n\n> Table 5. Overlapping evaluation - regression.\n\n| Model: MuMo          | ESOL (↓)   | Freesolv (↓) | LIPO (↓)   | QM7 (↓)    | QM8 (↓)    | QM9 (↓) |\n| -------------------- | ---------- | ------------ | ---------- | ---------- | ---------- | ------- |\n| Overlap Rate         | 37.67%     | 29.28%       | 6.90%      | 4.61%      | 2.09%      | 0.58%   |\n| Original Performance | 0.536      | 1.082        | 0.448      | 42.80      | 0.0111     | 0.00300 |\n| Clean Performance    | 0.342      | 0.796        | 0.439      | 45.67      | 0.0112     | 0.00300 |\n| Performance Change   | **+36.2%** | **+26.43%**  | **+2.01%** | **+6.70%** | **+0.90%** | 0.00%   |\n\n---\n\n**[Q1]** Contribution of Mamba vs. Fusion Scheme\n\nTo disentangle the contributions of the backbone versus the fusion scheme, we conducted an ablation study using the same PI + SFP across different backbones.\n\nShown in Table 6, **MuMo (Attn_Mamba)** achieves the best overall performance. In contrast:\n\n- **Mamba-Only (w/o fusion)** suffers a **59.55% performance drop**, confirming that the fusion module is critical, especially on regression tasks  (Caco-2, PPBR).\n- **MuMo (Transformer)** and **MuMo (Vanilla Mamba)**, while both using the same fusion scheme, perform worse than our full model.\n\nThese results suggest that the gains are not from Mamba alone, but from our **co-designed fusion strategy and attention-injected backbone**.\n\n> Table 6. Ablation on backbones.\n\n| Model                       | BBB (↑)   | PGP (↑)   | Caco2 (↓) | PPBR (↓)  | Avg. Drop |\n| --------------------------- | --------- | --------- | --------- | --------- | --------- |\n| **MuMo (Attn_Mamba, ours)** | **0.899** | **0.942** | **0.315** | **7.324** | **0.00%** |\n| MuMo (Transformer)          | 0.867     | 0.899     | 0.317     | 8.489     | 6.66%     |\n| MuMo (Vanilla Mamba)        | 0.869     | 0.908     | 0.396     | 7.465     | 8.65%     |\n| Mamba-Only (w/o fusion)     | 0.813     | 0.876     | 0.832     | 11.54     | 59.55%    |\n\n---\n\n**[Q2]** Robustness to Conformer Variability\n\nWe compare MuMo with both 2D and 3D baselines on QM. MuMo achieves SOTA performance on QM8 and QM9 (Table 7), and near to UniMol on QM7, demonstrating strong geometry modeling capabilities. \n\n> Table 7. Extended Benchmark on QM7/8/9 datasets.\n\n| Model       | QM7 (↓)  | QM8 (↓)    | QM9 (↓)     |\n| ----------- | -------- | ---------- | ----------- |\n| GROVER      | 92.0     | 0.0224     | 0.00986     |\n| DMPNN       | 103.5    | 0.0190     | 0.00814     |\n| AttentiveFP | 72.0     | 0.0179     | 0.00812     |\n| UniMol      | **41.8** | 0.0156     | 0.00467     |\n| MuMo        | 42.8     | **0.0111** | **0.00300** |\n\nThen, we evaluated MuMo under three conformer settings: 1) MMFF94-minimized, 2) UFF-minimized, 3) pure ETKDG. **MuMo maintains stable performance**, and exhibits less performance fluctuations than UniMol (Table 8), which confirms that our method is robust to 3D perturbations. \n\n> Table 8. Evaluation using different conformer optimization tools.\n\n| Task          | Model    | MMFF94    | UFF       | No-Optimize | Std. Dev ↓ | Max Δ ↓   |\n| ------------- | -------- | --------- | --------- | ----------- | ---------- | --------- |\n| ESOL  (↓)     | UniMol   | 0.769     | 0.790     | 0.939       | 0.0927     | 0.170     |\n| **ESOL  (↓)** | **MuMo** | **0.536** | **0.550** | **0.585**   | **0.0252** | **0.049** |\n| BBBP (↑)      | UniMol   | 0.889     | 0.831     | 0.733       | 0.0789     | 0.156     |\n| **BBBP (↑)**  | **MuMo** | **0.962** | **0.952** | **0.941**   | **0.0105** | **0.021** |\n\nTable 9 evaluates by perturbing atom positions with increasing levels of Gaussian noise. We observe that **UniMol’s performance degrades more noticeably** as the noise scale increases, while **MuMo remains  higher robustness to 3D geometry noise**. This confirms that MuMo is better suited for practical scenarios where generated conformers may be imprecise.\n\n> Table 9. Ablation on Conformer Noise.\n\n| Model - Dataset      | 0Å        | 0.05Å     | 0.1 Å     | 0.2Å      | Std. Dev ↓ | Max Δ ↓   |\n| -------------------- | --------- | --------- | --------- | --------- | ---------- | --------- |\n| UniMol - ESOL (↓)    | 0.769     | 0.765     | 0.760     | 0.831     | 0.0334     | 0.071     |\n| **MuMo - ESOL (↓)**  | **0.536** | **0.530** | **0.540** | **0.544** | **0.006**  | **0.014** |\n| UniMol - BBBP  (↑)   | 0.889     | 0.878     | 0.890     | 0.787     | 0.0496     | 0.103     |\n| **MuMo - BBBP  (↑)** | **0.962** | **0.960** | **0.961** | **0.954** | **0.0036** | **0.008** |"}, {"review_id": "ywubo4ntjS", "reviewer": "Reviewer_31yj", "summary": "This paper aims to better integrate 1D, 2D, and 3D molecular data to achieve more effective molecular modeling and improve the performance of downstream molecular property prediction tasks. Its central argument is that previous works adopted symmetric fusion strategies, which overlooked the impact of noise in 3D molecular structures. Therefore, the authors propose an asymmetric approach that progressively injects 2D and 3D information into the 1D representation. This strategy leads to improved performance.", "questions": "Beyond the weaknesses discussed above, I also have several specific questions:\n\nQ1: What is the exact difference between BACE-R and BACE-S tasks?\n\nQ2: Why are the reported results for Uni-Mol inconsistent with those in the original paper? For example, ClinTox performance is reported as 91.9 in the original work, but 81.8 here. Could you provide some evaluation details to clarify this discrepancy?\n\nQ3: Could you evaluate your model on QM7 and QM8 under the Uni-Mol setting? These datasets are relatively small and can be tested during the rebuttal period. They are indeed crucial for assessing the model’s ability to capture 3D structural information.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n1, The authors recognize that different molecular modalities carry inherently different types of information and thus argue against symmetric fusion strategies. This insight is novel and potentially inspiring for researchers working in this area.\n\n2, The paper provides extensive experimental details, ablation studies, and analyses, validating the effectiveness of almost every module in this complex model.\n\n**Weaknesses:**\nMy main concern lies in the conceptually questionable claims and flawed design choices:\n\n1, Incorrect motivation: The paper aims to address two major challenges: **conformer dependency** and **modality collapse** (Lines 30–58). However, both are misrepresented.\n\nFor **conformer dependency**, the authors state that “conformers generated by tools like RDKit often differ significantly in local arrangement even with the same molecule.” I must remind the authors that a single SMILES string **naturally** corresponds to multiple 3D conformers, and the probability of each conformer is energy-dependent. This is not an artifact of the tool. Moreover, different conformers can indeed exhibit different physicochemical properties (e.g., chirality-related behaviors).\n\nFor **modality collapse**, the paper gives an example where the embeddings of Ibuprofen and Ketoprofen are too close. However, when comparing only two specific objects, it is not meaningful to define whether their embeddings are “close” or “far.” Moreover, both drugs are NSAIDs, so it is reasonable that their embeddings are somewhat similar. Additionally, the experimental section fails to demonstrate whether modality collapse is effectively addressed.\n\n2, Methodological flaw: The paper introduces 3D information by constructing a 3D graph (Lines 126–133), but only incorporates bond lengths and bond angles (Line 131). But, in comparison to a 2D graph, the most significant additional information from a 3D conformation is the dihedral angle. Given known atom types and bond types (i.e., the SMILES or 2D graph), bond lengths and angles can often be easily inferred, while dihedral angles are far more complex and informative. Neglecting dihedral modeling severely weakens the claim that this is a 3D-aware model.\n\n3, Missing critical ablation study: A key motivation of the paper is the importance of asymmetric fusion. Thus, an ablation study comparing symmetric vs. asymmetric fusion strategies should be included.", "was_revised": true, "final_justification": "The three weaknesses I raised have all been well addressed, so I will raise my score to 4.", "rebuttal": "Thanks the reviewer for recognizing our asymmetric, progressive fusion approach and comprehensive experiments. Comments on conformer dependency and dihedral angles led us to add new experiments, further demonstrating MuMo’s geometric strength and robustness.\n\nTo save space, standard deviations are omitted below.\n\n**[W1a]** Conformer dependency is not a tool artifact\n\nWe thank and fully agree with the reviewer that a SMILES corresponds to multiple conformers, and that conformer differences can lead to distinct properties. \n\n`Clarifying \"Conformer Noise\"`: We use this term for the variability among conformers of the same molecule in **tasks with low conformer-dependence** (BBBP, screening). In these situations, accurate conformers are costly, such dependent can degrade **3D-reliant models**.  Hence our aim is to **reduce heavy reliance on explicit 3D** and deliver predictions robustness.\n\n` Two experiments to solidify our motivation.   ` \n\n1. **Conformer source sweep (Table 1)** – Switching among **MMFF94, UFF, and no optimization** makes UniMol drift (Std Dev ≈ 0.08–0.09), whereas **MuMo stays within ≤ 0.03**.\n\n> Table 1. Evaluation on ESOL and BBBP datasets using different conformer optimization tools. \n\n| Task     | Model    | MMFF94    | UFF       | No-Optimize | Std. Dev ↓ | Max Δ ↓   |\n| -------- | -------- | --------- | --------- | ----------- | ---------- | --------- |\n| ESOL (↓) | UniMol   | 0.769     | 0.790     | 0.939       | 0.0927     | 0.170     |\n| ESOL (↓) | **MuMo** | **0.536** | **0.550** | **0.585**   | **0.0252** | **0.049** |\n| BBBP (↑) | UniMol   | 0.889     | 0.831     | 0.733       | 0.0789     | 0.156     |\n| BBBP (↑) | **MuMo** | **0.962** | **0.952** | **0.941**   | **0.0105** | **0.021** |\n\n2. **Gaussian noise test (Table 2)** – As atomic noise grows (0-0.2Å), UniMol’s error spreads, while MuMo barely moves.\n\n> Table 2. Robustness to 3D Conformer Noise.\n\n| Model - Dataset      | 0Å        | 0.05Å     | 0.1 Å     | 0.2Å      | Std. Dev ↓ | Max Δ ↓   |\n| -------------------- | --------- | --------- | --------- | --------- | ---------- | --------- |\n| UniMol - ESOL (↓)    | 0.769     | 0.765     | 0.760     | 0.831     | 0.0334     | 0.071     |\n| **MuMo - ESOL (↓)**  | **0.536** | **0.530** | **0.540** | **0.544** | **0.006**  | **0.014** |\n| UniMol - BBBP  (↑)   | 0.889     | 0.878     | 0.890     | 0.787     | 0.0496     | 0.103     |\n| **MuMo - BBBP  (↑)** | **0.962** | **0.960** | **0.961** | **0.954** | **0.0036** | **0.008** |\n\n---\n\n**[W1b]** Modality collapse example is inconclusive\n\nWe clarify that **Figure 1 is not intended to support the modality collapse claim, but to illustrate the issue of conformer dependency**. \n\n`Motivation Clarification.` **DimeNet’s excessive conformer sensitivity:** Figure 1 in paper shows multiple conformers of Ibuprofen embed farther apart than an Ibuprofen–Ketoprofen pair, implying DimeNet over-weights geometry and can miss 2D-structure/identity differences.\n\n`Extended experiments done.` We repeated the analysis on five more molecules, and observed the same pattern. An updated figure will be included in the revised paper.\n\n`Motivation on modality collapse.` The collapse we refer to occurs in **symmetric fusion**, where poor alignment leads the model to over-rely on one modality. This is a known issue in multimodal literature[1], and we further validate our asymmetric design in **[W3]** .\n\n[1] Liu S, Wang H, Liu W, Lasenby J, Guo H, Tang J. Pre-training molecular graph representation with 3d geometry. arXiv preprint arXiv:2110.07728. 2021 Oct 7.\n\n---\n\n**[W2]** Missing dihedral angle weakens 3D modeling claim\n\nWe deeply thank and agree that dihedral (=torsion) angles encode critical geometric variation, and we have carefully addressed this concern step by step.\n\n`STEP 1` Are dihedral angles truly missing?\n\nA torsion (dihedral) angle $\n\\phi_{ijkl}\n  = \\operatorname{atan2}\\bigl(\n        (\\mathbf r_{ji}\\times\\mathbf r_{jk})\n        \\cdot\n        \\mathbf r_{kl},\\;\n        (\\mathbf r_{ji}\\times\\mathbf r_{jk})\n        \\cdot\n        (\\mathbf r_{jk}\\times\\mathbf r_{kl})\n    \\bigr),$ with  $\\mathbf r_{ab}=\\mathbf x_b-\\mathbf x_a , \\mathbf x_a=(x_a,y_a,z_a)$, is completely determined by the six inter-atomic distances $\\lVert\\mathbf r_{ij}\\rVert,\\dots,\\lVert\\mathbf r_{kl}\\rVert$.  Our graph already supplies **all pairwise distances within a 6 Å cutoff** and message passing runs ≥ 3 layers, an $i\\to l$ path that closes the $i-j-k-l$ quadrangle exists: the network can **implicitly reconstruct** $\\phi_{ijkl}$ without an explicit torsion term.\n\n`STEP 2` Why not add explicit torsion anyway?\nTo quantify the benefit we performed an explicit-torsion ablation:\n\n* Added $[\\sin\\phi_{ijkl},\\cos\\phi_{ijkl}]$ on every rotatable bond.  \n* Kept all other hyper-parameters unchanged.\n\nTable 3&4 shows minimal difference on QM9 property benchmarks, but adding it causes memory and runtime to grow sharply. **This suggests that adding explicit torsion yields only marginal gains, while incurring notable computational overhead.**\n\n> Table 3. Explicit-torsion ablation on QM9 (↓).\n\n| Task (MAE)        | w/o Torsion | w/ Torsion | Δ (%) |\n| ----------------- | ----------- | ---------- | ----- |\n| alpha (↓)         | 0.283       | 0.281      | +0.71 |\n| Cᵥ (↓)            | 0.126       | 0.113      | +10.3 |\n| HOMO/LUMO/GAP (↓) | 0.0030      | 0.0032     | -6.67 |\n| **Average**       |             |            | +1.00 |\n\n> Table 4. Efficiency comparison. Values are in \"Pretrain/SFT(QM8)\" format with batch size 128/64.\n\n| Model variant     | GPU Mem (GB) ↓ | Step time (ms) ↓ |\n| ----------------- | -------------- | ---------------- |\n| Ours (implicit)   | **44.4/14.4**  | **987/731**      |\n| +Explicit torsion | 64.6/29.4      | 2623/1316        |\n\n`STEP 3` Good performance on conformer sensitivity benchmark\n\nAs discussed in **[Q3]**, we conducted evaluations on QM7/8/9.\n\n`STEP 4`  Conclusion\n\nThese experiments demonstrate that our model effectively captures geometric information **without incurring unnecessary overhead from over-engineering explicit structures**.\n\n---\n\n**[W3]** Suggest ablation: symmetric vs. asymmetric fusion\n\n**MuMo-FG**: fixed 2/3D encoder w/o joint modeling.  \n\n**MuMo-CE/CB**: concat modalities at the end or beginning.\n\nResults in Table 5 confirm the advantage of the asymmetric design.\n\n> Table 5. Ablation on fusion approches.\n\n| Model    | Type           | BBB (↑)   | PGP (↑)   | Caco2 (↓) | PPBR (↓)  | Avg. Drop |\n| -------- | -------------- | --------- | --------- | --------- | --------- | --------- |\n| **MuMo** | **Asymmetric** | **0.899** | **0.942** | **0.315** | **7.324** | **0.00%** |\n| MuMo-FG  | Asymmetric     | 0.878     | 0.901     | 0.412     | 7.935     | 11.96%    |\n| MuMo-CE  | Symmetric      | 0.889     | 0.917     | 0.475     | 8.273     | 16.88%    |\n| MuMo-CB  | Symmetric      | 0.849     | 0.891     | 0.601     | 9.132     | 31.60%    |\n\n---\n\n**[Q1]** Clarification on BACE-R vs. BACE-S tasks\n\nBACE-R and BACE-S denotes the data using Random split and Scaffold split (MoleculeNet Recommendation).\n\n---\n\n**[Q2]** Clarification on discrepancy in Uni-Mol results - Clintox Dataset\n\nFrom Table 6, the average deviation is even **+0.88%**. BBBP even has an +21.5% performance.\n\nThe larger gap on ClinTox and BBBP stems from extreme imbalance(Table 7). Importantly, our model MuMo outperforms by a large margin (**Avg. +16.2%**).\n\n> Table 6. Difference of Uni-Mol results.\n\n| Dataset                 | BBBP (↑) | BACE (↑) | ClinTox (↑) | Tox21 (↑) | SIDER (↑) | ESOL (↓) | FreeSolv (↓) | Lipo (↓) | Δ to Original |\n| ----------------------- | -------- | -------- | ----------- | --------- | --------- | -------- | ------------ | -------- | ------------- |\n| UniMol - Original Paper | 72.9     | 85.7     | 91.9        | 79.6      | 65.9      | 0.788    | 1.480        | 0.603    | 0.0%          |\n| UniMol - Our Reports    | 88.6     | 84.0     | 81.8        | 81.2      | 66.6      | 0.769    | 1.598        | 0.597    | +0.88%        |\n| MuMo                    | 95.7     | 84.9     | 98.5        | 83.4      | 67.7      | 0.536    | 1.082        | 0.448    | +16.2%        |\n\n> Table 7. Data distribution from BBBP and Clintox.\n\n| Dataset | Positive Class Ratio | Negative Class Ratio |\n| ------- | -------------------- | -------------------- |\n| BBBP    | 83%                  | 17%                  |\n| ClinTox | 7%                   | 93%                  |\n\n---\n\n**[Q3]** Request for evaluation on QM7 and QM8\n\nWe include **QM7/8/9** in our evaluation (Table 8) and even on QM9 following the **official Uni-Mol-v2 split** for fair comparison (Table 9).\n\n`Experiment Results` Our model outperforms **outperforms Uni-Mol-v2 on 6/8 tasks**, with half model size and (505M) and less pretrain data (1/400).\n\n> Table 8. Evaluation on QM7/8/9 from MoleculeNet.\n\n| Model       | QM7 (↓)  | QM8 (↓)    | QM9 - HOMO/LUMO/GAP (↓) |\n| ----------- | -------- | ---------- | ----------------------- |\n| GROVER      | 92.0     | 0.0224     | 0.00986                 |\n| DMPNN       | 103.5    | 0.0190     | 0.00814                 |\n| AttentiveFP | 72.0     | 0.0179     | 0.00812                 |\n| UniMol      | **41.8** | 0.0156     | 0.00467                 |\n| **MuMo**    | 42.8     | **0.0111** | **0.00300**             |\n\n> Table 9. Evaluation on QM9 from Uni-Mol-v2.\n\n| Model         | HOMO/LUMO/GAP (↓) | alpha (↓) | Cᵥ (↓)    | mu (↓)    | R² (↓)    | ZPVE (↓)   |\n| ------------- | ----------------- | --------- | --------- | --------- | --------- | ---------- |\n| GEM           | 0.0067            | 0.589     | 0.237     | 0.444     | 25.67     | 0.0011     |\n| Uni-Mol2 570M | 0.0036            | 0.315     | 0.147     | 0.089     | 4.523     | 0.0005     |\n| Uni-Mol2 1.1B | 0.0035            | 0.305     | 0.144     | **0.089** | **4.265** | 0.0005     |\n| **MuMo 505M** | **0.0030**        | **0.283** | **0.126** | 0.400     | 18.08     | **0.0005** |"}, {"review_id": "l5Ch8VzzXA", "reviewer": "Reviewer_7PGe", "summary": "The authors identify two critical challenges in multimodal molecular representation learning: the unreliability of 3D conformer data, which can be noisy and inconsistent, and modality collapse, where naive fusion methods allow noisy signals from 3D geometry to degrade information from 2D topology or SMILES. To address these issues, the authors propose MuMo, introducing Structured Fusion Pipeline (SFP) and  Progressive Injection (PI). SFT integrates 2D topological graphs and 3D geometric information into a unified, stable structural prior. PI  asymmetrically injects this stable structural prior into a separate sequence-based model (operating on SMILES strings) at later stages of the network. Evaluated on 21 benchmark tasks from Therapeutics Data Commons (TDC) and MoleculeNet, MuMo demonstrates state-of-the-art performance on 17 tasks.", "questions": "- Among the evaluated benchmarks, the classification benchmarks appear to have limited performance room, and the molecule distribution seems restricted. For more conclusive experimental evaluation, could the authors consider assessing their method on benchmarks with diverse molecular distributions and greater performance room, such as PubChemQC?\n- The Structured Fusion Pipeline (SFP) is designed to create a stable structural prior to mitigate 3D conformer noise. How does MuMo's performance degrade as the quality of the input 3D conformers decreases?\n- The authors use a Mamba backbone, for its efficiency and ability to handle long-range dependencies. At the same time, the authors state that their core contributions (SFP and PI) are model-agnostic. Have the authors experimented with or can they speculate on how MuMo would perform with a more traditional Transformer backbone?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "### Strengths\n\n- Clear Problem Formulation and Compelling Motivation: The paper presents a compelling motivation by illustrating the two core problems in multimodal molecular representation learning—conformer instability and modality collapse (Figure 1).\n- Comprehensive Evaluation: The authors validate MuMo across an extensive set of 21 benchmark tasks, comparing it against a wide range of strong baselines, including 2D graph-based, 3D-aware, and large-scale pretrained models. The model achieves state-of-the-art results on a majority of the tested benchmarks.\n- Novel and Promising Architecture: The proposed fusion strategy is novel and successfully aggregates promising components. The concept of first creating a stable, unified structural prior (SFP) before injecting it into the main sequence stream appears to be a promising approach to handling noisy inputs. The Progressive Injection (PI) mechanism is a thoughtful solution to modality collapse, moving beyond simplistic early-fusion or symmetric attention schemes.\n\n### Weaknesses\n\n- Architectural Complexity: The MuMo framework is quite complex, integrating three separate data streams (2D, 3D, sequence), multiple message-passing neural networks (MPNNs), a state-space backbone (Mamba), and a custom cross-attention injection mechanism. This complexity may pose challenges for reproducibility, point-to-point ablation for each component of the proposed method.\n- Dependence on 3D Conformer Generation: While the SFP is designed to *reduce* sensitivity to conformer quality, the model still requires 3D geometric inputs. The framework does not eliminate the computational cost and potential pipeline dependencies associated with generating these conformers in the first place. The paper could benefit from a more explicit analysis of performance trade-offs with varying levels of conformer quality.\n- Limited Discussion on Computational Overhead: The paper focuses heavily on prediction performance but provides limited discussion on the computational trade-offs (e.g., inference time, memory usage) of its complex architecture compared to the baselines, especially simpler 2D or sequence-only models. This information would be valuable for assessing its practical applicability.", "was_revised": true, "final_justification": "The author's rebuttal effectively resolved my concerns on 3D conformer generation and computational overhead. Reading the author's rebuttal, I am convinced that the framework shows good performance with significantly less computational burden than the prior works, while incorporating 3D modality effectively.", "rebuttal": "We sincerely thank the reviewer for recognizing our motivation to address **conformer instability** and **modality collapse**, and for highlighting our **novel fusion design**, **stable structural prior**, and **extensive 21-task evaluation**. We also appreciate the concerns on **model complexity** and **3D reliance**, which we address below.\n\n**[W1]** Model complexity and difficulty of reproducibility\n\nThank you for raising this concern. We acknowledge that our design incorporates fine-grained fusion **across sequences, 2D, and 3D,** while also addressing **global-local** representation needs and the **limitations of symmetric fusion**. This led to our use of SFP and the PI mechanism.\n\nMoreover, MuMo is implemented as a fully **encapsulated, end-to-end framework**:\n\n- It can take a single SMILES string as input and automatically handles 2D/3D conformer generation and structural segmentation via RDKit. \n- It also supports preprocessed multimodal inputs. The usage is user-friendly: our open-sourced package includes complete training scripts,  all the data, model weights, and **a one-line command** to reproduce all results. \n- In response to this feedback, we have also done the codebase with the latest features used during rebuttal, which will be released after double-blind review.\n\n`Comprehensive ablations.` We recognize the difficulty of conducting ablation studies on a complex system and have accordingly provided well-structured ablation experiments in the paper (Sec. 4.3, Appendix D), along with additional experiments during the whole rebuttal phase.\n\n---\n\n**[W2]** Dependence on 3D conformers without quality sensitivity analysis\n\n`Thanks and agree.` Thank you for pointing out the reliance on 3D conformers. We fully agree that conformer generation introduces computational overhead. Specifically, we attempted to generate 3D conformers for 10M molecules and found it took approximately 2 weeks, while 2D processing completed within 1 hour. \n\n`Ablation without 3D.` However, our model is not strictly dependent on 3D inputs. **MuMo (1D-2D) achieves competitive results compared to baselines on most tasks (Table 1)**. This flexibility allows users to trade off accuracy and efficiency based on resource constraints. \n\n`Ablation on low quality 3D.` Additionally, we conducted experiments by adding different levels of noise to 3D coordinates (Table 2) to explicitly evaluate MuMo’s robustness, which **confirms its resilience to 3D quality degradation**.\n\n> Table 1. Ablation Study: Performance without 3D Inputs. Top-2 are bolded.\n\n| Method        | BBBP  (↑)           | ClinTox  (↑)        | ESOL (↓)            | LIPO (↓)            |\n| ------------- | ------------------- | ------------------- | ------------------- | ------------------- |\n| Best Baseline | **0.949 (0.013)**   | 0.933 (0.054)       | 0.617 (0.077)       | 0.597 (0.034)       |\n| MuMo (1D)     | 0.931 (_0.005_)     | 0.958 (_0.021_)     | 1.793 (_0.055_)     | 0.844 (_0.033_)     |\n| MuMo (1D-2D)  | 0.946 (_0.008_)     | **0.971 (_0.015_)** | **0.597 (_0.051_)** | **0.596 (_0.035_)** |\n| MuMo (1D-3D)  | **0.957** (_0.011_) | **0.985** (_0.011_) | **0.536** (_0.061_) | **0.577** (_0.027_) |\n\n> Table 2 Robustness to 3D Conformer Noise.  Top-2 are bolded.\n\n| Model - Dataset      | noise_scale - 0Å  | noise_scale - 0.05Å | noise_scale - 0.1 Å | noise_scale - 0.2Å | Std. Dev ↓ | Max Δ ↓   |\n| -------------------- | ----------------- | ------------------- | ------------------- | ------------------ | ---------- | --------- |\n| UniMol - ESOL (↓)    | 0.769 (0.153)     | 0.765 (0.144)       | 0.760 (0.141)       | 0.831 (0.220)      | 0.0334     | 0.071     |\n| **MuMo - ESOL (↓)**  | **0.536 (0.061)** | **0.530 (0.069)**   | **0.540 (0.040)**   | **0.544 (0.071)**  | **0.006**  | **0.014** |\n| UniMol - BBBP  (↑)   | 0.889 (0.025)     | 0.878 (0.032)       | 0.890 (0.031)       | 0.787 (0.039)      | 0.0496     | 0.103     |\n| **MuMo - BBBP  (↑)** | **0.962 (0.007)** | **0.960 (0.007)**   | **0.961 (0.008)**   | **0.954 (0.010)**  | **0.0036** | **0.008** |\n\n---\n\n**[W3]** Missing discussion of computational overhead\n\nThank you for highlighting the lack of discussion on computational overhead. We have added detailed comparisons to address this concern. \n\n`High efficient in pretraining.` As shown in Table 3, MuMo is highly efficient in the pretraining stage, **requiring only 5 hours** on a 1.6M-scale dataset using 4×A100 GPUs, significantly less than prior models that consume days or weeks but got better performance.\n\n`High efficient in finetuning.` In the finetuning stage (Table 4), even on our **largest** benchmark dataset QM9 (133k samples), MuMo completes training in **just 3.3 hours** with modest GPU memory (14G per card). For smaller datasets (few thousand samples), training typically finishes in under 30 minutes. \n\n`Conclusion.` These results demonstrate that MuMo offers strong performance without prohibitive resource demands, making it practical for real-world use.\n\n> Table 3. Computing Cost Comparation in Pretraining Stage.\n\n| Model          | Scale  | Pretrain Dataset Size | Pretrain Time             |\n| -------------- | ------ | --------------------- | ------------------------- |\n| MoLFormer-Base | 44.28M | 1.1B                  | 16×V100-32G, Est. 104h    |\n| MoLFormer-XL   | 86.75M | 1.1B                  | 16×V100-32G, 208h         |\n| UniMol         | 47.61M | 19M                   | 4×V100-32G, 20h           |\n| UniMol-v2      | 1.1B   | 884M                  | 64×A100-80G, Est. 2 weeks from its paper|\n| MuMo           | 505M   | 1.6M                  | 4×A100-80G, 5h            |\n\n> Table 4. Computing Cost on QM9 in Finetuning Stage.\n\n| Item                 | Value                           |\n| -------------------- | ------------------------------- |\n| Dataset              | QM9                             |\n| Sample Count         | 133,855                         |\n| CPU Cores            | 32                              |\n| CPU                  | Intel(R) Xeon(R) Platinum 8480+ |\n| GPU Count            | 2                               |\n| GPU Type             | NVIDIA A100-SXM4-80GB           |\n| Model Parameters     | 505M                            |\n| Total FLOPs          | 65,218,081 GF                   |\n| Epochs               | 10                              |\n| Global Batch Size    | 64                              |\n| Max GPU Memory Usage | 14G per GPU                     |\n| Data Preprocess Time | 5 min                           |\n| Training Time        | 3h 21m 51s                      |\n\n---\n\n**[Q1]** Limited benchmark diversity; propose PubChemQC\n\n`Computational prohibition.` Thank you for suggesting PubChemQC. We agree it is a large and chemically diverse quantum chemistry dataset (~3.8M molecules). However, due to its sheer scale, it is widely used for **pretraining** (e.g., UniMol-v2, MolFormer-XL), but **rarely used as a benchmark**, as fully evaluating multiple baselines would be **computationally prohibitive.**\n\n`Comprehensive benchmarks.` In contrast, our evaluation spans **21 standard benchmarks** from **MoleculeNet** and **TDC** (extended to 28 in rebuttal stage), covering both classification and regression. These include diverse tasks such as **toxicity (e.g., Tox21, SIDER, ClinTox), pharmacokinetics (e.g., BBBP, Pgp, HIA, Bioavailability), bioactivity (BACE), and physical chemistry (e.g., ESOL, FreeSolv, Lipo)**. Among them, **17/21 are classification tasks**, and MuMo consistently outperforms strong 2D, 3D, and pretrained models.\n\nWe agree that scaling MuMo to even larger and more diverse datasets like PubChemQC is a valuable direction. We are actively working toward this, while maintaining efficient inference for real-world use.\n\n---\n\n**[Q2]** Robustness under low-quality 3D conformers\n\nThank you for raising this point again. We have addressed this in detail under **[W2]**\n\n---\n\n**[Q3]** Generalizability to Transformer backbones\n\n`Ablation on backbone selection.` Thank you for the thoughtful question. To evaluate the generality of our fusion scheme, we conducted controlled experiments using the same SFP + PI design across different backbone architectures. As shown in Table 5, **MuMo (Transformer)** and **MuMo (Vanilla Mamba)** both perform reasonably well, but consistently underperform compared to our full model using the **hybrid Attn-Mamba backbone**.\n\n`Ablation on fusion.` We also include two backbone-only baselines without fusion: Mamba-only and Transformer-only. These models show substantial drops in performance, up to **59.55%**  (mostly on regression tasks), confirming that the **core performance gains come from our co-designed fusion strategy (SFP + PI)**.\n\n`Conclusion.` These results support our claim that the fusion design is model-agnostic and effective across architectures. Nonetheless, its full potential is best realized when combined with our lightweight attention-enhanced backbone that balances expressiveness and efficiency.\n\n> Table 5 Ablation on Backbone Generalizability.\n\n| Model                          | BBB (↑)           | PGP (↑)           | Caco2 (↓)         | PPBR (↓)          | Avg. Drop |\n| ------------------------------ | ----------------- | ----------------- | ----------------- | ----------------- | --------- |\n| **MuMo (Attn_Mamba, ours)**    | **0.899 (0.014)** | **0.942 (0.019)** | **0.315 (0.055)** | **7.324 (0.323)** | **0.00%** |\n| MuMo (Transformer)             | 0.867 (0.051)     | 0.899 (0.049)     | 0.317 (0.095)     | 8.489 (0.591)     | 6.66%     |\n| MuMo (Vanilla Mamba)           | 0.869 (0.011)     | 0.908 (0.020)     | 0.396 (0.045)     | 7.465 (0.333)     | 8.65%     |\n| Mamba-Only (w/o fusion)        | 0.813 (0.019)     | 0.876 (0.023)     | 0.832 (0.075)     | 11.54 (0.788)     | 59.55%    |\n| MuMo (Transformer, w/o fusion) | 0.843 (0.021)     | 0.889 (0.013)     | 0.644 (0.077)     | 9.892 (0.701)     | 37.84%    |"}, {"review_id": "94yD8Zbfzg", "reviewer": "Reviewer_5uq6", "summary": "This paper proposes MuMo, a multimodal molecular representation learning model that combines 2D topology, 3D geometry, and SMILES sequence information. To address conformer instability and modality collapse, the authors introduce a Structured Fusion Pipeline (SFP) to construct a stable unified graph, and a Progressive Injection (PI) mechanism to inject structural priors into a state-space sequence model. Experiments show consistent performance improvements, with MuMo ranking first on 17 tasks.", "questions": "Together with the above weaknesses, I have the following questions，\n\n1. Have you tried replacing Mamba with a Transformer? Does the injection mechanism still help?\n2. Could you add chirality-aware features like dihedral angles to address the enantiomer limitation?\n3. Is it possible to extend MuMo for generative tasks like molecule synthesis or scaffold decoration?", "limitations": "The authors have discussed the limitations.", "rating": 4, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1. The architecture directly addresses known issues in multimodal molecular learning, including conformer variance and modality dominance.\n2. Ablation studies are thorough and well-organized, demonstrating the impact of each design component.\n\n**Weaknesses**\n\n1. The model is tightly integrated with the Mamba backbone, but its generality with other architectures is untested.\n2. The writing is dense in places, with overly technical notation that could be simplified.", "was_revised": true, "final_justification": "I think the authors' detailed responses and additional experiments have addressed my concerns, but it does not significantly shifted my overall impression. I will keep my score.", "rebuttal": "We sincerely thank the reviewer for the positive and detailed feedback. We're especially grateful for the recognition of our **clear architectural motivation**, **innovative fusion strategy**, **comprehensive ablations**, and **strong performance across diverse benchmarks**. We also appreciate the thoughtful suggestions that we address below.\n\n**[W1]** Generalizability to other backbones\n\nThank you for this point. To explicitly assess this, we conducted controlled experiments using the same SFP + PI fusion strategy across multiple backbones. \n\n`Ablation on backbone.` As shown in Table 1, **MuMo with Transformer** and **Vanilla Mamba** backbones both achieve competitive results, confirming that our fusion design is indeed **model-agnostic**.\n\n`Ablation on fusion.` In contrast, when the fusion module is removed, both **Mamba-only** and **Transformer-only** baselines suffer large performance drops, which are up to **59.55%** and **37.84%**, respectively. This highlights that the **key performance gains come from our structured fusion strategy**, not from the choice of backbone alone.\n\nOverall, while our hybrid Attn-Mamba design yields the strongest results, our ablations demonstrate that the proposed fusion scheme can generalize across architectures, validating its standalone effectiveness.\n\n> Table 1. Ablation on Backbone Generalizability.\n\n| Model                          | BBB (↑)           | PGP (↑)           | Caco2 (↓)         | PPBR (↓)          | Avg. Drop |\n| ------------------------------ | ----------------- | ----------------- | ----------------- | ----------------- | --------- |\n| **MuMo (Attn_Mamba, ours)**    | **0.899 (0.014)** | **0.942 (0.019)** | **0.315 (0.055)** | **7.324 (0.323)** | **0.00%** |\n| MuMo (Transformer)             | 0.867 (0.051)     | 0.899 (0.049)     | 0.317 (0.095)     | 8.489 (0.591)     | 6.66%     |\n| MuMo (Vanilla Mamba)           | 0.869 (0.011)     | 0.908 (0.020)     | 0.396 (0.045)     | 7.465 (0.333)     | 8.65%     |\n| Mamba-Only (w/o fusion)        | 0.813 (0.019)     | 0.876 (0.023)     | 0.832 (0.075)     | 11.54 (0.788)     | 59.55%    |\n| MuMo (Transformer, w/o fusion) | 0.843 (0.021)     | 0.889 (0.013)     | 0.644 (0.077)     | 9.892 (0.701)     | 37.84%    |\n\n---\n**[W2]** Overly dense writing and technical notation\n\nThank you for the helpful suggestions regarding clarity and conciseness. We acknowledge that some parts of the main paper and appendix were overly dense or verbose. Specifically:\n\n- In the **main paper**, Sections 3.1–3.3 (lines 104–157) included compact notation and stacked equations (e.g., multi-modal alignment, fusion operators, injection formulas) that could hinder readability. We have revised these sections to simplify the notation, add explanatory cues, and restructure overly technical sentences.\n- In the **appendix**, certain subsections in **Appendix B** were longer than necessary. In particular, the descriptions of model variants and extended ablation settings repeated parts of the main text and included redundant detail. We will streamline these texts to retain essential insights while improving clarity and conciseness.\n\nWe thank the reviewer again for highlighting these issues and have prepared the updated content for final version accordingly.\n\n---\n\n**[Q1]** Generalizability to transformer backbones\n\nThank you for bringing this quesiton again, we have addressed it in **[W1]**.\n\n---\n\n**[Q2]** Incorporating chirality-aware features (e.g., dihedral angles)\n\nWe thank the reviewer for highlighting this important point. We agree that dihedral (torsion) angles encode critical geometric variation, especially for flexible molecules, and we have carefully addressed this concern step by step:\n\n`STEP 1` Are dihedral angles truly missing?\n\nA torsion (dihedral) angle $\n\\phi_{ijkl}\n  = \\operatorname{atan2}\\bigl(\n        (\\mathbf r_{ji}\\times\\mathbf r_{jk})\n        \\cdot\n        \\mathbf r_{kl},\\;\n        (\\mathbf r_{ji}\\times\\mathbf r_{jk})\n        \\cdot\n        (\\mathbf r_{jk}\\times\\mathbf r_{kl})\n    \\bigr),$ with  $\\mathbf r_{ab}=\\mathbf x_b-\\mathbf x_a , \\mathbf x_a=(x_a,y_a,z_a)$, is completely determined by the six inter-atomic distances $\\lVert\\mathbf r_{ij}\\rVert,\\dots,\\lVert\\mathbf r_{kl}\\rVert$.  Because our graph already supplies **all pairwise distances within a 6 Å cutoff** and message passing runs ≥ 3 layers, an $i\\to l$ path that closes the $i-j-k-l$ quadrangle exists: the network can **implicitly reconstruct** $\\phi_{ijkl}$ without an explicit torsion term.\n\n`STEP 2` Why not add explicit torsion anyway?\nTo quantify the benefit we performed an **explicit-torsion ablation**:\n\n* Added $[\\sin\\phi_{ijkl},\\cos\\phi_{ijkl}]$ on every rotatable bond.  \n* Kept all other hyper-parameters unchanged.\n\nTable 2&3 shows ≤ 1.1 % difference on QM9 property benchmarks, but adding torsion explicitly cause finetuning memory usage grows × 2.1 and runtime × 1.8 (Table 3). **This suggests that adding explicit torsion yields only marginal gains, while incurring notable computational overhead.**\n\n> Table 2. Explicit-torsion ablation on QM9 (mean ± s.d., ↓) .\n\n| Task (MAE)        | w/o Torsion    | w/ Torsion     | Δ (%)     |\n| ----------------- | -------------- | -------------- | --------- |\n| alpha (↓)         | 0.283 (0.003)  | 0.281 (0.003)  | +0.71     |\n| Cᵥ (↓)            | 0.126 (0.003)  | 0.113 (0.003)  | +10.3     |\n| ZPVE (↓)          | 0.0005 (6e-05) | 0.0005 (6e-05) | 0.0       |\n| HOMO/LUMO/GAP (↓) | 0.0030 (1e-05) | 0.0032 (1e-05) | -6.67     |\n| **Average**       |                |                | **+1.01** |\n\n> Table 3. Efficiency comparison. Values are in Pretrain/SFT(QM8) format with global batch size 128/64.\n\n| Model variant     | GPU Mem (GB) ↓ | Step time (ms) ↓ |\n| ----------------- | -------------- | ---------------- |\n| Ours (implicit)   | **44.4/14.4**  | **987/731**      |\n| +Explicit torsion | 64.6/29.4      | 2623/1316        |\n\n`STEP 3` Good performance on conformer sensitivity benchmark\n\nAs Table 4 shows, we conducted evaluations on QM7, QM8, and QM9. The results confirm that **MuMo effectively captures 3D-dependent properties** and exhibits strong robustness across conformer-sensitive tasks.\n\n> Table 4. Evaluation on QM7/8/9 benchmarks from MoleculeNet.\n\n| Model       | QM7 (↓)        | QM8 (↓)             | QM9 - HOMO/LUMO/GAP (↓) |\n| ----------- | -------------- | ------------------- | ----------------------- |\n| GROVER      | 92.0 (0.9)     | 0.0224 (0.0003)     | 0.00986 (0.00025)       |\n| DMPNN       | 103.5 (8.6)    | 0.0190 (0.0001)     | 0.00814 (0.00001)       |\n| AttentiveFP | 72.0 (2.7)     | 0.0179 (0.0010)     | 0.00812 (0.00001)       |\n| UniMol      | **41.8 (0.2)** | 0.0156 (0.0001)     | 0.00467 (0.00004)       |\n| **MuMo**    | 42.8 (0.6)     | **0.0111 (0.0001)** | **0.00300 (0.00001)**   |\n\n`STEP 4`  Changes prepared to update for paper\n\n1. **Section 3.2.1 (**Unified Graph**)** – add:  \n   “Given a 6 Å distance graph, multi-layer message passing allows implicit reconstruction of dihedral angles (Added Appendix A.3).”\n2. **Section 4.2** - Evaluation on QM7/8/9  (Table 4).  \n3. **Appendix D.8** – insert the new ablation results (Table 2).  \n4. **Appendix D.9** – insert the comparison. (Table 3).\n\nThese additions demonstrate that our model effectively captures geometric information **without incurring unnecessary overhead from over-engineering explicit structures**.\n\n`STEP 5` Conclusion\n\n* The network implicitly encodes dihedral information via multi-hop distance features. \n* Adding explicit torsions yields marginal (< 1.1 %) gain at > 2× computational cost.\n* On a torsion-only benchmark we still outperform an explicit-torsion baseline.\n\nWe appreciate the reviewer’s insight and have incorporated these clarifications and new experiments into the revised manuscript.\n\n---\n\n**[Q3]** Possible to extend to molecule synthesis or scaffold decoration tasks?\n\n`Explaination.` Thank you for the interesting question. MuMo is currently designed as an **encoder-only** model, in line with the baselines we compare against, which focus on molecular representation learning. Tasks such as molecule generation or scaffold decoration typically require **encoder–decoder architectures**, and fall under a different modeling paradigm.\n\n`Future work.` So, we agree that the strong performance of MuMo in representation tasks suggests **promising potential in generative settings**. Extending our multimodal fusion framework to support molecular generation is a direction we are actively exploring in our next work.\n\n**[Formatting Concerns]** Appendix placement before checklist\n\nWe confirm that our submission follows the NeurIPS formatting guidelines—specifically, the appendix is correctly placed **before** the checklist section."}]

Author rebuttal:

Meta-review: This paper proposes MuMo, a multimodal molecular representation learning framework that addresses two key challenges: instability due to unreliable 3D conformers and modality collapse in naive fusion methods. The authors introduce a Structured Fusion Pipeline (SFP) to integrate 2D topology and 3D geometry into a stable structural prior and a Progressive Injection (PI) mechanism to asymmetrically inject this prior into a sequence backbone (Mamba-based). The core claim is that this design enhances robustness to 3D noise and prevents modality dominance. Extensive experiments verify the effectiveness of the proposed methods.

The paper makes a significant contribution to molecular representation learning with novel insights, strong results, and practical utility. The rebuttal resolved all concerns convincingly, warranting acceptance as a poster. The work is not nominated for oral/spotlight due to its incremental architectural advances and the limited technical novelty rather than transformative novelty.
Decision: Accept (poster)