SMI-Editor: Edit-based SMILES Language Model with Fragment-level Supervision

Authors: Kangjie Zheng, Siyue Liang, Junwei Yang, Bin Feng, Zequn Liu, Wei Ju, Zhiping Xiao, Ming Zhang
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
Keywords: SMILES Language Model, SMILES Pre-training Model, Molecular Pre-training Model

Abstract:
SMILES, a crucial textual representation of molecular structures, has garnered significant attention as a foundation for pre-trained language models (LMs). However, most existing pre-trained SMILES LMs focus solely on the single-token level supervision during pre-training, failing to fully leverage the substructural information of molecules. This limitation makes the pre-training task overly simplistic, preventing the models from capturing richer molecular semantic information. Moreover, during pre-training, these SMILES LMs only process corrupted SMILES inputs, never encountering any valid SMILES, which leads to a train-inference mismatch. To address these challenges, we propose SMI-Editor, a novel edit-based pre-trained SMILES LM. SMI-Editor disrupts substructures within a molecule at random and feeds the resulting SMILES back into the model, which then attempts to restore the original SMILES through an editing process. This approach not only introduces fragment-level training signals, but also enables the use of valid SMILES as inputs, allowing the model to learn how to reconstruct complete molecules from these incomplete structures. As a result, the model demonstrates improved scalability and an enhanced ability to capture fragment-level molecular information. Experimental results show that SMI-Editor achieves state-of-the-art performance across multiple downstream molecular tasks, and even outperforming several 3D molecular representation models.

Decision: accept (poster)
Reviewer scores: 5, 8, 8, 6, 3

Reviews and discussion:
<thread>
creation date:
1729586432057

thread_type:
official_review

<thread_object>
creator: reviewer_agu3

<content>
summary:
In this paper, the authors propose SMILES-Editor, an edit-based pre-trained SMILES language model by introducing a new pre-training strategy that randomly corrupts SMILES strings during the pre-training process and lets the LLM restore the original SMILES strings. Experimental results show that the new proposed pre-training strategy achieves better performance than the previous MLM task in various downstream tasks.

strengths:
1. The idea of focusing on the atoms and fragments is meaningful and enables better learning of SMILES representations.
2. The new pre-training task may lead to an improvement in the valid generation of molecule SMILES representations.
3. The SMILES-Editor shows competitive performance compared to the original MLM, bringing an improvement in the pre-training of SMILES Language Models.

weaknesses:
1. The authors claim that "most existing pre-trained SMILES language models (LMs) only provide supervision at the single-token level during pre-training and fail to fully leverage substructural information of molecules", which lacks enough support. In works like  [1], the tokenizers are initialized with SMILES Pair Encoding. Meanwhile, the sentence-piece method can also find common fragments in molecule SMILES strings, and these common fragments are usually critical functional groups. In this case, I do not agree with this claim.
2. The novelty of SMILES-Editor is limited. The main contribution of this paper is to propose a new pre-training strategy, while this strategy is not novel, and there is not a significant difference compared to the previous Masked Span Language Modeling used by T5. Furthermore, more advanced LLMs are now using decoder-only structures, while SMILES-Editor can only be adopted in LLMs with encoder structures, which further harms the novelty.
3. The experiments on the classification tasks do not incorporate the k-fold experiments, which raises concerns about the stability of the method.
4. The experiment results do not seem satisfying enough. And the comparison should include more pre-training strategies.
5. In Table 2, the performance of SMI-EDITOR-AtomsMasking in BACE is even better than SMILES-Editor.

#### References
[1] Li, X., & Fourches, D. (2021). SMILES pair encoding: a data-driven substructure tokenization algorithm for deep learning. Journal of chemical information and modeling, 61(4), 1560-1569.

questions:
1. Could the authors find more evidence to support their claims in Weakness 1?
2. Could the authors explain more about Weakness 2?
3. I am wondering about the comparison between different pre-training strategies. Could the authors compare the auto-regressive pre-training strategies in identical decoder-only structures?
4. What is the time cost or complexity of SMILES-Editor compared to the previous MLM?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730096725746

thread_type:
official_review

<thread_object>
creator: reviewer_rnmw

<content>
summary:
The paper introduces SMI-Editor, a novel edit-based SMILES language model with fragment-level supervision to improve molecular representation learning. SMI-Editor tries to address the limitations of masked language models (MLMs), such as rapid saturation and limited substructure semantics modeling, by using an LevT-based modeling.

strengths:
- The analysis identifies interesting issues: the MLM model struggles to distinguish between random deletion and hydrophilic deletion, and it quickly saturates on single-token masking.
- The paper writing is clear and well-illustrated.

weaknesses:
- LevT isn’t actually an MLM model; it’s a non-autoregressive generative model. From this perspective, the authors' solution for handling corrupted SMILES input is somewhat trivial, as other sequence-based generative models could also address this.
- There is a discrepancy between the motivation and solution. Neither corrupted SMILES nor fragment-level supervision is directly related to the way LevT models molecules.
- Limited effectiveness. While I understand the authors' choice of the MoleculeNet baseline without considering powerful graph-based models like Uni-Mol, DVMP_{TF} in Jinhua Zhu et al. is a SMILES-based Transformer encoder and appears to perform much better than the reported results.
- Figure 5 is confusing; if the model can distinguish hydrophilic groups, wouldn’t performance degrade noticeably after HG deletion?

questions:
- Why not include the performance on FreeSolv and ESOL?
- What about applying MLM on fragment spans? This could also help alleviate the saturation problem in single-token prediction.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730279108855

thread_type:
official_review

<thread_object>
creator: reviewer_sia5

<content>
summary:
This paper proposes an edit-based pre-trained SMILES language model to learn 3D molecular representations and applies the model to several downstream tasks, such as molecular property prediction. The idea of using token edit operations to replace token masking introduces a novel approach with the potential to address the rapid saturation problem. The paper provides comprehensive details on model implementation and includes extensive ablation studies.

strengths:
Novelty: The use of token edit operations instead of traditional token masking is innovative and could offer significant advantages in terms of model performance and stability.
Comprehensive Implementation: The paper provides detailed descriptions of the model architecture and training procedures.
Extensive Ablation Studies: The authors have conducted thorough ablation studies to validate the effectiveness of their approach.

weaknesses:
1. Figure 4: The requirement for an expert to provide training signals is inefficient and limits the size of the training set. This could be a significant drawback in practical applications.
2. Figure 5: Figures 3 and 5 should be presented together for easier comparison. Additionally, the figure's purpose is not clearly explained.
3. Figure 6: The figure's ability to demonstrate scalability is questionable. It would be helpful to show whether larger models yield better results to substantiate claims of scalability.
4. Conclusion: The statement, "ablation studies confirm the advantages of its design over traditional MLMs in modeling molecular substructure semantics and training stability," is not fully supported by the evidence presented in Figure 5. The figure does not convincingly demonstrate that the model understands molecular substructure semantics.

questions:
1. Figure 3: The purpose of this figure is unclear. It would be beneficial to include a comparison with other methods to highlight the advantages of the proposed approach.
2. Table 1: It is important to specify the amount of data used for fine-tuning to better understand the model's performance.
3. Generative Tasks: It would be interesting to explore whether this model can be applied to generative tasks, which could broaden its applicability.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730726328748

thread_type:
official_review

<thread_object>
creator: reviewer_mxjn

<content>
summary:
The paper first investigates the problem of MLM training, as train-inference mismatch. To address this, the authors propose the edit-based pre-trained language model, SMI-Editor, for SMILES. As another pre-training strategy, SMI-Editor drops the substructures in a valid SMILES sequence and tries to recover the original molecules.. The extensive experiments demonstrate the effectiveness of SMI-Editor.

strengths:
1. The model is well-motivated to solve the train-inference mismatch problem in existing MLM paradigm.

2. Extensive experiments indicate the effectiveness of SMI-Editor.

3. The analysis of the rapid saturation problem of MLM and how SMI-Editor can solve this problem is generally convincing.

4. The paper is well-written and easy to follow.

weaknesses:
1. The authors put a lot efforts to compared SMI-Editor with MLM paradigm. However, there are still other strategies for pre-training the model, for example, contrastive learning. Can the authors also discuss the relations between SMI-Editor and other pre-training paradigms?

2. As the authors state, increasing the mask ratios will influence the convergence in MLM. Would fragment drop ratio also influences the SMI-Editor?

questions:
See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730812948530

thread_type:
official_review

<thread_object>
creator: reviewer_uwik

<content>
summary:
This paper identifies three key challenges in existing SMILES masked language models: the neglect of substructural information, overly simplistic training tasks, and a mismatch between training and inference procedures and introduce a SMILES language model employing edit-based, fragment-level supervision to address these challenges.

strengths:
1.	This paper reveals the shortcomings of existing SMILES masked language models through an analysis of experimental results, thereby informing future research directions.
2.	This paper presents a novel SMILES language model that uses edit-based, fragment-level supervision. This approach improves performance on the molecule property prediction task.

weaknesses:
1.	Edit-based models, as currently designed, focus solely on restoring removed substructures during pre-training. Other valuable sources of molecular information, such as correcting errors and removing extraneous components, may be neglected.
2.	The data processing and pre-training stages lack clarity. A detailed case study would significantly improve understanding of these processes.
3.	The description of the experimental tasks is inconsistent between the caption of Table 1 and the " 5.2 RESULTS ON MOLECULAR PROPERTY PREDICTION" section.

questions:
1.	How to ensure that left-over fragments can be assembled into a valid molecule? Please provide an exact example.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731987038765

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Reponses (Part 1)

comment:
# General Response (Part 1)

We would like to express our sincere gratitude to all the reviewers for dedicating their time to thoroughly evaluate our work. We are delighted to receive valuable suggestions from the reviewers. We appreciate **uWiK** and **SiA5** for acknowledging the innovation of using edit-based, fragment-level supervision in SMI-Editor; **mXJn** and **Rnmw** for recognizing the well-motivated design and insightful analysis of the train-inference mismatch and rapid saturation problems; **SiA5** and **AGu3** for acknowledging the detailed implementation and significant performance improvements of SMI-Editor on molecular property prediction and valid SMILES generation; and **mXJn** and **Rnmw** for commending the clarity and readability of the paper. We sincerely thank all the reviewers for their insightful comments and constructive feedback.

**We have also carefully addressed all concerns and questions raised by each reviewer.** Incorporating the feedback, we have conducted additional experiments to strengthen our explanations and support our claims. All new results, analyses, and revisions have been included in the updated version of the paper. In addition, all new images, example codes, and other files supplemented during the rebuttal stage have been packaged and uploaded to the supplementary materials. They can be downloaded and viewed directly if needed. **If further clarifications are needed, please do not hesitate to let us know.**

Before addressing specific questions and concerns, we would like to share two significant new experimental findings that may interest the reviewers:

P.S: Due to the inclusion of a substantial number of new results during this rebuttal stage, we have labeled all tables in the rebuttal as Table A, Table B, etc., to avoid conflicts with the original paper's numbering (e.g., Table 1, Table 2).


## Part 1:Excellent Performance of SMI-Editor on DeepChem Data

We re-evaluated the performance of SMI-Editor on various downstream tasks of MoleculeNet benchmark using the data splits provided by DeepChem. Previously, our experiments were based on a different data split, which made it difficult to compare our model against others built on this dataset. Therefore, we re-tested SMI-Editor on DeepChem splits and included comparisons with more baseline models. Detailed results are presented in Table A1. As shown in Table A1, **SMI-Editor achieves significant performance gains over baseline models, reaching state-of-the-art levels with noticeable average performance improvements.** We have also added these resuts to Appendix I in the revised paper. Below is a detailed analysis of these results:

1. **SMI-Editor outperforms models trained with various paradigms:** On average, SMI-Editor surpasses molecular representation learning models like MolCLR and DMP_{TF}, which use contrastive pretraining, as well as models like ChemBerta and SMI-MLM, which use masked language modeling. It also outperforms autoregressive language models like Galactica and graph-based models like MolCLR, MGSSL, and MoMu. These results highlight the potential of SMILES language models.

2. **SMI-Editor achieves competitive performance with less training data:** SMI-Editor outperforms DMP_{TF}, which is trained on over 100 million compounds, despite using only 19 million compounds for training. This demonstrates SMI-Editor's higher data efficiency, enabled by its ability to effectively leverage substructure information from SMILES sequences.

Table A1. Overall results on MoleculeNet datasets using DeepChem splits ([DeepChem GitHub](https://github.com/deepchem/deepchem)). ROC-AUC scores (higher is better) are reported for all tasks. The best results are **bolded**.

|            | BBBP↑    | Tox21↑   | ClinTox↑ | HIV↑     | BACE↑    | SIDER↑   | Mean↑    |
| ---------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- |
| GEM        | 72.4     | 78.1     | 90.1     | 80.6     | 85.6     | 67.2     | 79.0     |
| ChemBerta  | 64.3     | -        | 90.6     | 62.2     | -        | -        | -        |
| MolCLR     | 73.6     | 79.8     | 93.2     | 80.6     | 89.0     | 68.0     | 80.7     |
| MGSSL      | 70.5     | 76.5     | 80.7     | 79.5     | 79.7     | 61.8     | 74.8     |
| DMP_{TF}   | 78.1     | 78.8     | 95.0     | 81.0     | 89.3     | 69.2     | 81.9     |
| Galactica  | 66.1     | 68.9     | 82.6     | 74.5     | 61.7     | 63.2     | 69.5     |
| MoMu       | 70.5     | 75.6     | 79.9     | 76.2     | 77.1     | 60.5     | 73.3     |
| SMI-MLM    | 89.4     | 76.2     | 90.6     | 79.8     | 86.6     | 66.5     | 81.5     |
| SMI-Editor | **93.5** | **81.4** | **95.2** | **81.6** | **89.9** | **69.8** | **85.2** |
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces SMI-Editor, an edit-based pre-trained SMILES language model aimed at improving molecular representation learning. The authors propose a novel pre-training strategy, which involves randomly corrupting SMILES strings by dropping substructures and training the model to restore the original molecules, providing fragment-level supervision and better leveraging molecular substructure semantics.

Overall, this paper is well motivated and well written. By using edit-based, fragment-level supervision instead of traditional token masking, SMI-Editor effectively tackles the rapid saturation and train-inference mismatch problems. The authors provide the comprehensive experiments to validate the effectiveness of the proposed pre-training strategy. 

Meanwhile, some concerns were raised regarding the clarity of data processing and pre-training stages, suggesting that detailed explanations and case studies would enhance understanding.  Additionally,  the literature review of this paper is not sufficient. It dose not include the broader pre-training strategies of molecules, including molecular graph pre-training and contrastive pre-training approaches.  

Minor issue: at Line 82, RELATD should be RELATED. 

In a summary,  The technical novelty is somehow limited. Extending the mask strategy to the fragment-level is a minor improvement. But I still recommend the acceptance of this paper based on its comprehensive experiments.