MV-CLAM: Multi-View Molecular Interpretation with Cross-Modal Projection via Language Model

Authors: Sumin Ha, Jun Hyeong Kim, Yinhua Piao, Sun Kim
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Molecule captioning, large language models, drug discovery, molecule representation learning

Abstract:
Large language models (LLMs) have shown significant potential in the biomolecular domain, particularly by demonstrating that effective adaptation of molecular representations for LLMs can greatly improve the quality of molecular captions. Most previous works have focused on aligning unimodal molecular structures with text, overlooking the diversity of modalities. Naive approaches to aligning multi-modal molecular structures with text often lead to (1) separately aligned embeddings, (2) inconsistent textual representations, and (3) increased computational overhead. To address these challenges, we propose LLM framework MV-CLAM equipped with MQ-Former, a novel multi-querying transformer. This architecture introduces a cross-model projector facilitating the simultaneous alignment of 2D and 3D molecular representations to a unified text token. By employing a shared self-attention layer, MQ-Former preserves rich molecular embeddings across different dimensions while consolidating them into a universal molecular token. Our approach outperforms baseline models in both molecule-text retrieval and molecule captioning tasks. Additionally, our framework shows promising results for zero-shot molecule editing and molecule-related question answering. By effectively integrating multi-view molecular data into a format conducive to LLMs, our method serves as a valuable tool for enhancing the characterization and understanding of chemical structures, facilitating a more seamless transition from molecular data to textual descriptions. The source code of MV-CLAM is available in https://anonymous.4open.science/r/mv-clam-4827.

Decision: reject
Reviewer scores: 3, 5, 5, 3

Reviews and discussion:
<thread>
creation date:
1730282154723

thread_type:
official_review

<thread_object>
creator: reviewer_r1xc

<content>
summary:
The paper proposes MQ-Former, an extension of the Q-Former framework, which incorporates a multi-query mechanism for aligning both 2D and 3D molecular data with textual information for enhanced molecule-text retrieval and molecule captioning.

strengths:
- The paper aims to enhance cross-modal alignment by integrating 2D and 3D molecular views.
- The model demonstrates improvements in molecule-text retrieval and captioning performance over baseline models.
- The paper includes case studies and examples of zero-shot molecule editing.

weaknesses:
- The model lacks significant innovation, as MQ-Former primarily adds an extra branch to the existing Q-Former with only minor variations in training objectives.
- Experiments are restricted to molecule-text retrieval and captioning on PubChem. The paper lacks essential molecular tasks like molecule generation and datasets like ChEBI-20.
- The motivation for adding a branch to Q-Former, rather than simply using a 3D molecular encoder like prior works (e.g., 3D-MoLM), is unclear. 
- The paper’s presentation could be improved. Plots lack careful formatting, with text that is difficult to read due to small font sizes.

questions:
- How does MQ-Former handle scenarios where 2D and 3D molecular information may not equally contribute to textual descriptions?
- Could the authors include more molecular tasks, such as molecule generation or property prediction, to provide a more comprehensive evaluation of MQ-Former?
- What impact does the weighting of the multi-objective training loss have on the model’s performance?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730284339594

thread_type:
official_review

<thread_object>
creator: reviewer_48rx

<content>
summary:
The paper introduces MV-CLAM, a framework utilizing a novel multi-querying transformer (MQ-Former) to enhance the alignment of multi-modal molecular representations with text. By employing a shared self-attention layer, this approach effectively consolidates 2D and 3D molecular data into query tokens, improving performance in molecule-text retrieval and captioning tasks. Additionally, it demonstrates potential for zero-shot molecule editing and molecule-related question answering, thereby facilitating better characterization of chemical structures.

strengths:
* The description of the proposed methodology is easy to follow. The paper is well written in general.
* The paper introduces a promising multi-view for approach for the infusion of specialized chemical knowledge into general-purpose pre-trained LLMs.
* The proposed MV-CLAM achieves state-of-the-art on PubChem324K for molecule captioning and retrieval tasks.

weaknesses:
* The experimental evaluation of the proposed method is conducted on a single dataset for both task: molecule captioning and molecule-text retrieval.
* The list of baseline models on molecule captioning only includes a single T5 language model while there are more recent works, including: nach0 and Text+ChemT5. 
* Some implementation decisions are not justified well enough. This includes: (i) the choice of SciBERT as a language encoder for MQ-Former; (ii) the choice of 2D and 3D encoders; (iii) introduction of $K$ query tokens instead of a single query token for each view; (iv) the choice of LLaMA2 as an LLM. It is unclear how the experimental results would change if each of the mentioned models is replaced with another one.
* Incomplete ablation study. The necessity of (i) Molecule-text Contrasting and (ii) Molecule-text Matching losses is not proven experimentally. For (i), it is unclear whether two loss components required or the model will perform well with a single one. For (ii), the impact of negative sample is under-explored. 
* The effect of most hyper-parameters in the method's module on the resulting performance is understudied. For instance, query token count, negative sample count in MTM loss.
* The methodology for molecule-text retrieval is unclear from the paper.
* The applicability of the proposed methodology to broader list of datasets is questionable: it requires 2D/3D molecular data in addition to simple SMILES string representations.

questions:
* Add experimental comparison against more chemical language models on molecule captioning, e.g., nach0 [1], Text+Chem T5 [2], SciFive [3], PRESTO [4], GitMol [5].
* For retrieval task (Table 1), is it possible to add chemical BERT-based encoders in addition to textual encoder SciBERT? (e.g., ChemBERTa)
* Conduct additional experiments on other molecule captioning datasets such as Mol-Instructions [6] and CheBI20 [7].
* For molecule-text retrieval, do you adopt a generative approach (e.g., GENRE [8]) or the task is formulated as a cross-modal embedding-based search by similarity (e.g., as in [9])?
* In Figure 3, where does the textual description come from during prediction on a test set? As far as I understand the molecule captioning task, you are only given a SMILES string.
* What is the LLaMA version you use? Add adopted HuggingFace checkpoints. 
* Even if you adopt a LLaMA with 7B parameters, MolT5 has less than 1B. Could not we just scale MolT5 to 3-5B parameters and obtain a better molecule captioning quality?
* Why is MolT5 absent from the Table 1?
* Add ablation study for SciBERT, 2D/3D molecule encoders, LLaMA2.
* Add ablation study for training losses. For Molecule-text Contrasting loss, prove it requires two components. For Molecule-text Matching loss, explore the effect of negative samples.
* Is it possible to generalize the methodology to unseen datasets and unseen SMILES? Given a SMILES, can I always obtain its 2D/3D representation and apply a pre-trained MV-CLAM model?




Typos:
* Line 102: transformer -> Transformer, Add reference.
* Line 194: **$A$** under-specified.
* Line 234: Missing citation for LoRA.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730557631368

thread_type:
official_review

<thread_object>
creator: reviewer_vfcg

<content>
summary:
The paper introduces a framework that leverages large language models (LLMs) to interpret and generate molecular captions. The work incorporates both 2D and 3D molecular structures to provide a more comprehensive understanding of molecules.

strengths:
1. The paper integrates both 2D and 3D molecular structures to enhance the model's understanding of molecular data.
2. The paper includes detailed figures (Figure 1-3) that clearly explain the method's framework and training scheme. 
3. And the analysis of attention maps in Appendix A.4 provides valuable insights into the model's behavior.

weaknesses:
1. Compared to recent related work, such as 3D-MoLM (Li et al., 2024), the innovation in MV-CLAM appears incremental. While the paper claims to incorporate both 2D and 3D molecular structures for a more comprehensive understanding, the approach seems to merely extend the 3D-MoLM framework by introducing 2D components through MAT. The proposed MQ-former architecture does not demonstrate significant structural innovations beyond existing methods. A clearer articulation of the novel contributions and architectural advantages over 3D-MoLM would be necessary to establish the work's originality.
2. The paper considers SMILES as an important molecular modality and notes that "1D SMILES provide compact represen tation of molecular structures", but does not mention SELFIES (Krenn et al., 2020) at all, which has been widely adopted in recent works due to its robust characteristics and tokenization-friendly nature. SELFIES offers inherent robustness and easier tokenization that aligns well with LLMs, making it a potentially more suitable choice for this application. 
3. Some images (e.g. the big image at page 18) are not vector graphics and lack titles or captions, which makes it confusing.

questions:
See 'Weaknesses' section.
1. Could the authors provide a more detailed explanation of the novelty of MV-CLAM compared to recent related work?
2. Why was SELFIES not considered as a molecular modality in this work, given its advantages over SMILES in tokenization and alignment with LLMs?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730656252504

thread_type:
official_review

<thread_object>
creator: reviewer_6jay

<content>
summary:
The work proposes a novel multimodal LLM framework MV-CLAM for organic chemistry and MQ-Former — multi-querying transformer model for simultaneous 1D, 2D, and 3D molecular representation learning. Authors show SOTA results in two tasks of molecule-text retrieval and molecule captioning. In addition, authors claim that their approach allows zero-shot molecule editing and molecule-related question answering.

strengths:
New molecular multimodal LLM framework for simultaneous incorporation of 1d 2D and 3D representations.
New Transformer architecture MQ-Former.

weaknesses:
The claim of the state-of-the-art performance for molecule captioning is not satisfied, see the results in [6].
There is no comparison with the other strong retrieval methods for the molecule retrieval task, i.e. RAG.
There are various problems with the Zero-shot editing part of the paper. The task is not formally defined. There are no metrics nor baselines for it.

The QA part is practically absent in the paper, while claimed in the abstract and results parts..
There are many works on molecular conformation generation [1-4], it seems that SMILES and/or 2D-graph representation is enough for neural networks to reconstruct RDKIT conformations almost perfectly. It means that 3D input possibly does not add any new information to the model. There is no comparison of the 1D+2D+3D MQ-Former vs 1D+2D models in the paper.

There is no comparison with other works on multi-modal representation learning for molecules, e.g.: [5]. 

[1] Zhu, Jinhua, et al. "Direct Molecular Conformation Generation."
[2] Xu, Minkai, et al. "GeoDiff: A Geometric Diffusion Model for Molecular Conformation Generation." International Conference on Learning Representations.
[3] Jing, Bowen, et al. "Torsional diffusion for molecular conformer generation." Advances in Neural Information Processing Systems 35 (2022): 24240-24253.
[4] Lee, Danyeong, et al. "Disco: Diffusion Schrödinger bridge for molecular conformer optimization." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 12. 2024.
[5] Manolache, Andrei, Dragos Tantaru, and Mathias Niepert. "MolMix: A Simple Yet Effective Baseline for Multimodal Molecular Representation Learning." arXiv preprint arXiv:2410.07981 (2024).
[6] Liu, Zhiyuan, et al. "ReactXT: Understanding Molecular" Reaction-ship" via Reaction-Contextualized Molecule-Text Pretraining." arXiv preprint arXiv:2405.14225 (2024).

questions:
1. 3D structures (conformers)

As mentioned in sec. 5.1 you use MMFF for molecular conformation generation.

a. Is it ETKDG geometry generation with further MMFF optimization?
b. Since it is possible to generate several different conformers for a single molecular structure, did you assess the dependence of the model quality on the conformations? Is it necessary to optimize a generated with ETKDG conformer with MMFF?

2.  It would be reasonable to compare your approach for Zero-shot editing with conditional generation models for small molecules.

3. Please, add experiments on the CHEBI-20 benchmark for the captioning task.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732439441090

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response (1). Concerning SOTA baseline models - 1

comment:
We sincerely appreciate the constructive feedback regarding the comparison with state-of-the-art (SOTA) models. **We chose 3D-MoLM [1] and UniMoT [2] as our baseline models because they are the most closely aligned with our model in terms of structural similarity and training data, allowing for the most equitable evaluation regarding our novel cross-modal projector, MQ-Former.** We appreciate the suggestion to implement additional SOTA methods and have carefully considered their inclusion.

While additional comparisons with other SOTA methods remain important, there are differences in dataset composition and preprocessing pipelines across suggested models (Table R1). Even within the PubChem dataset, prior models adopted different preprocessing procedures, leading to variations in the number of molecule entities. As a result, previous works also have demonstrated inconsistencies in the selection of baseline models. **To ensure a fair comparison, we aligned our selection with UniMoT, the most recent model trained on the same dataset.** We updated Table 1 in the revised manuscript - presented as Table R2 - to include MoleculeSTM [3] and MolCA [4]. Similarly, Table 2 in the revised manuscript - shown as Table R3 - was updated to include InstructMol [5] and MolCA as baseline models, following the UniMoT paper.

We recognize the value of retraining other models under the same dataset condition for a rigorous comparison. However, training a single model and performing additional fine-tuning would exceed the available timeline. Therefore, we concentrated our efforts on the most feasible and impactful evaluations within the given discussion timeframe. We prioritized validating the key contributions and novelty of our proposed MQ-Former model through ablation studies and targeted experiments that justify its design and effectiveness. These experiments highlight the distinct advantages of our approach in leveraging 2D and 3D molecular representations to align with textual information as shown in Tables R4-R6, and the updated Appendix Figures 7 and 13 in the revised manuscript.



**Table R1.** Pretraining datasets of baseline models
| Model        | Dataset - #pretrain                         |
|--------------|---------------------------------------------|
| MolCA        | PubChem-MolCA-298k                         |
| 3DMoLM       | PubChem-3DMoLM-301k                        |
| UniMoT       | PubChem-3DMoLM-301k                        |
| MolBind [6]     | PubChem-MolBind-319k                       |
| TextChem T5 [7] | CheBI-Train                                |
| GitMol [8]      | PubChem+CheBI-GitMol-320k → 90k (multimodal)|
| Atomas [9]      | PubChem-Atomas-243k → 51k (high-quality, leak-free) |
| Presto [10]      | PubChem-Presto-326k                        |
| AMORE [11]       | CheBI-Train                                |
| SciFive [12]     | CheBI-Train                                |
| nacho [13]       | Mol-Instructions                           |
</content>
</thread_object>

</thread>
<thread>
creation date:
1732440019523

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response (2). Concerning the CheBI20 dataset

comment:
We appreciate the reviewer’s comments and acknowledge the importance of using diverse and extensive datasets for evaluating the generalizability of our model. In this work, we chose to validate our MV-CLAM framework for molecule captioning solely on the PubChem324k dataset, **following the baseline approach (3D-MoLM) we built upon**, which also reports captioning performances on the PubChem324k dataset. 

Our decision to exclude the ChEBI-20 dataset was primarily motivated by the following considerations:
1. Data Redundancy and Leakage Concerns
> ChEBI-20 is derived from PubChem324k dataset, with additional manual curation for specific biological contexts. Since ChEBI-20 is essentially a subset of PubChem, there is an inherent overlap between the two datasets. This overlap raises potential concerns about data redundancy and leakage when training and evaluating on these datasets together.

2. Evaluation of Molecular Nomenclature
> Unlike the PubChem324k dataset, which retains molecular names and provides a broader variety of molecular structures, ChEBI-20 replaces molecular names with generic placeholders such as “the molecule.” While this emphasizes molecular properties, it limits the evaluation of the model’s ability to connect structural features with accurate molecular nomenclature. Names often encode critical structural information (e.g., functional groups, stereochemistry, or ring systems), making them an essential aspect of evaluating a model’s understanding of molecular structures.

Taking these considerations into account, we believe that using the **PubChem dataset provides a rigorous and comprehensive evaluation of our framework’s capabilities in text retrieval, molecule captioning, and downstream tasks**. We have included a detailed discussion in **Appendix A.2** on our dataset selection process and the rationale for excluding ChEBI20. We emphasize that *validating our captioning performance on the PubChem dataset alone is sufficient, as it offers a significantly larger and more diverse set of molecular descriptions, enabling a robust assessment of the generalizability and efficacy of our model*. We deeply appreciate the reviewers for thoughtful suggestions. 


References
> [1] Li et al., "Towards 3D Molecule-Text Interpretation in Language Models". [2] Zhang et al., "UniMoT: Unified Molecule-Text Language Model with Discrete Token Representation" [3] Liu et al., "Multi-modal Molecule Structure-text Model for Text-based Retrieval and Editing" [4] Liu et al., "MolCA: Molecular Graph-Language Modeling with Cross-Modal Projector and Uni-Modal Adapter". [5] Cao et al., "InstructMol: Multi-Modal Integration for Building a Versatile and Reliable Molecular Assistant in Drug Discovery". [6] Xiao et al., "MolBind: Multimodal Alignment of Language, Molecules, and Proteins" [7] Christofidellis et al., "Unifying Molecular and Textual Representations via Multi-task Language Modelling" [8] Liu et al., "GIT-Mol: A Multi-modal Large Language Model for Molecular Science with Graph, Image, and Text" [9] Zhang et al., "Atomas: Hierarchical Alignment on Molecule-Text for Unified Molecule Understanding and Generation" [10] Cao et al., "PRESTO: Progressive Pretraining Enhances Synthetic Chemistry Outcomes" [11] Ganeeva et al., "Lost in Translation: Chemical Language Models and the Misunderstanding of Molecule Structures" [12] Phan et al., "SciFive: a text-to-text transformer model for biomedical literature" [13] Livne et al., "nach0: Multimodal Natural and Chemical Languages Foundation Model"
</content>
</thread_object>

</thread>
<thread>
creation date:
1732440207618

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response (3). Concerning novelty & technical contributions

comment:
We sincerely thank the reviewers for their valuable feedback regarding the novelty and technical contributions of MQ-Former, and we apologize for not clearly conveying these aspects in our initial submission. MQ-Former *aligns molecular and text spaces by sharing self-attention layers between each multi-view structural information (2D and 3D) and text*, generating a universal query token interpretable to LLMs. The **simultaneous yet separate alignment of 2D and 3D structural representation to text**, is only applicable with our novel MQ-Former, and ensures a balanced incorporation of  abundant structural information among the differing views. This design addresses key challenges in multi-modal learning to *align multiple modalities with minimal information loss*. We have demonstrated this through our case study of captions, attention visualization and ablation study of using single view molecular embeddings mentioned in Section 6.3. 

To further address your comments and demonstrate the uniqueness of our approach, we have conducted two additional studies: 1)  Embedding space visualization to demonstrate MQ-Former preserves modality-specific information in accordance to textual semantics. 2) Comparison ablation that utilizes multi-view molecular embeddings (2D+3D embeddings concatenated) in the former Q-Former framework. This necessitates the usage of employing MQ-Former with an additional branch, differentiating it from Q-Former. 

1. **Embedding Space Visualization**

> We analyzed the embeddings of the 2D, 3D queries, and our universal query tokens in the latent space alongside the corresponding word embeddings from textual descriptions. Specifically, we focused on highly 2D- or 3D-related words from the textual captions described in Appendix Figure 6-7 (Case Study 1. Attention visualization). 
For 2D-related words, the distance between the embeddings followed the trend: 2D < 2D+3D < 3D.Conversely, for 3D-related words, the trend was: 3D < 2D+3D < 2D.These observations confirm that MQ-Former successfully preserves modality-specific information while aligning it with textual semantics, highlighting the interplay between 2D and 3D molecular views in multi-modal learning. We have included the visualization results in **Appendix Figure 7**.

2. **Comparison with Multi-View Representations in Q-Former Framework**

> To highlight the necessity of MQ-Former, we conducted an ablation study comparing our architecture with a variant that aligns multi-view molecular representations using a single Q-Former module. The suggested MolMix [14] (a multimodal representation learning framework) by reviewer 6JAy lacks pretrained weights for molecular embeddings. Hence, we leveraged the 2D embeddings from MAT and the 3D embeddings from Uni-Mol, concatenating these representations before projecting them into the textual space using the Q-Former. We combined the results with the former ablation conducted in Table 3. 

> Overall, the study emphasizes that while single-view embeddings (e.g., 2D or 3D alone) capture important molecular information, they lack the comprehensive representation needed for captioning tasks requiring multi-faceted insights. Moreover, unlike the concatenation-based approach, MQ-Former preserves the rich, distinct representations of molecular views. That is, *the simultaneous alignment of these embeddings in the shared textual space enhances the preservation of intricate molecular properties*.  This design facilitates more fine-grained alignment with text, maintaining diversified information, which results in higher-quality captions across all evaluated metrics (Table R4). Overall, MQ-Former enables the preservation of detailed and diverse molecular representations, facilitating precise alignment with textual descriptions and delivering superior performance across captioning task.

**Table R4**. Comparison of molecule captioning performance: multi-view embeddings aligned with Q-Former
|                        | BLEU2  | BLEU4  | METEOR | ROUGE1 | ROUGE2 | ROUGE-L |
|----------------------------|--------|--------|--------|--------|--------|--------|
| **2D + Q-Former** | 29.72 | 22.26 | 34.22 | 38.22 | 23.45 |  31.61  |
| **3D+ Q-Former** | 29.45 | 22.03 | 33.79 | 37.86 |  23.11 | 31.83  |
| **Multi-view (2D&3D) + Q-Former** | 29.80  | 22.70  | 35.49  | 39.07  | 24.92  | 33.09  |
| **MV-CLAM**            | **31.75**  | **24.48**  | **36.54**  | **40.43**  | **25.72**  | **33.79**  |

This novelty ensures more precise molecule-text understanding, distinguishing *MQ-Former as a robust and effective solution for molecular captioning*. We appreciate the opportunity to clarify our contributions and have incorporated these findings into the revised manuscript organized in **Appendix A.4. Effectiveness of MQ-Former**.


Reference 
> [14] Manolache, Andrei, Dragos Tantaru, and Mathias Niepert. "MolMix: A Simple Yet Effective Baseline for Multimodal Molecular Representation Learning."
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes a novel multimodal LLM framework, MV-CLAM, for organic chemistry, and MQ-Former, a multi-query transformer model for simultaneous 1D, 2D, and 3D molecular representation learning, aiming to provide a more comprehensive understanding of molecules.

The idea of integrating 1D, 2D, and 3D molecular information is interesting. The design of MQ-Former involves adapting the Q-Former technique from multimodal learning. However, the experimental results are not convincing enough to demonstrate the superiority of the proposed method, due to issues such as missing baselines, insufficient ablation studies, and reliance on a single dataset. Additionally, the technical novelty is somewhat limited. Therefore, I do not recommend the acceptance of this paper.