SMI-Editor: Edit-based SMILES Language Model with Fragment-level Supervision
Authors: Kangjie Zheng
Siyue Liang
Junwei Yang
Bin Feng
Zequn Liu
Wei Ju
Zhiping Xiao
Ming Zhang
Venue: ICLR 2025 Poster (2025)
Keywords: SMILES Language Model
SMILES Pre-training Model
Molecular Pre-training Model
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)

Abstract: SMILES, a crucial textual representation of molecular structures, has garnered significant attention as a foundation for pre-trained language models (LMs). However, most existing pre-trained SMILES LMs focus solely on the single-token level supervision during pre-training, failing to fully leverage the substructural information of molecules. This limitation makes the pre-training task overly simplistic, preventing the models from capturing richer molecular semantic information. Moreover, during pre-training, these SMILES LMs only process corrupted SMILES inputs, never encountering any valid SMILES, which leads to a train-inference mismatch. To address these challenges, we propose SMI-Editor, a novel edit-based pre-trained SMILES LM. SMI-Editor disrupts substructures within a molecule at random and feeds the resulting SMILES back into the model, which then attempts to restore the original SMILES through an editing process. This approach not only introduces fragment-level training signals, but also enables the use of valid SMILES as inputs, allowing the model to learn how to reconstruct complete molecules from these incomplete structures. As a result, the model demonstrates improved scalability and an enhanced ability to capture fragment-level molecular information. Experimental results show that SMI-Editor achieves state-of-the-art performance across multiple downstream molecular tasks, and even outperforming several 3D molecular representation models.

Published as a conference paper at ICLR 2025
SMI-EDITOR: EDIT-BASED SMILES LANGUAGE MODEL WITH FRAGMENT-LEVEL SUPERVISION
Kangjie Zheng S Siyue Liang!.2. Junwei Yang!.23 Bin Feng4.1.2.8 Zequn Liu
Wei Ju Zhiping Xiao Ming Zhang"l.2. 22233
1 Sccoool of Computer Science, Peking University.
National Key Laboratory for Multimedia Information Processing, Peking University.
PPeking University-Anket Embodied AI Lab, Peking University.
*International Digital Economy Academy (IDEA), Shenzhen, China.
"College of Computer Science, Sichuan University, Chengdu, China.
'Computer Science and Engineering Department, University of Washington, U.S.A. juwei aww...du.cc... patxiaoQuu.edie mzhang.cs wwt.ing.ssppkuu.du..
ABSTRACT
SMILES, a crucial textual representation of molecular structures, has garnered sig- nificant attention foundation for pre-trained language models (LMs). However, most existing pre-trained SMILES LMs focus solely on the single-token level super- vision during pre-training. failing to fully leverage the substructural information of molecules. This limitation makes the pre-training task overly simplistic, preventing the models from capturing richer molecular semantic information. Moreover, dur- ing pre-training. these SMILES LMs only process corrupted SMILES inputs, never encountering any valid SMILES, which leads to a train-inference mismatch. To address these challenges, propose SMI-EDITOR, a novel edit-based pre-trained S.I.ES LM. SMI EMITOR disrupts substructures within a molecule at random and feeds the resulting SMILES back into the model, which then attempts to restore the original SMIL through an editing process. This approach not only introduces fragment-level training signals, but also enables the use of valid SMILES inputs, allowing the model to learn how to reconstruct complete molecules from these incomplete structures. As result, the model demonstrates improved scalability and an enhanced ability to capture fragment-level molecular information. Experimental results show that EMI-EDITOR achieves state-of-the-an art performance across mul- tiple downstream molecular tasks, and even outperforming several 3D molecular representation models.
INTRODUCTION
With the widespread application of Al in molecular-related tasks, enhancing the modeling of SMILES data has become key research focus. The textual nature of SMILES data makes possible for us to leverage past experiences from text modeling to address challenges in SMILES representation. Additionally, the knowledge extracted from SMILES data often aligns well with textual knowledge. A large number of studies have attempted to design SMILES language models (LMs) to explore the knowledge inherent in SMILES sequences (Wang et al. 2019a Chithrananda et al. 2020 Bagal et al.
202T Ross et al., 2022). Significant efforts have also been made to align learned know ledge from hewwwadd//ddwwrdss SMILES with textual knowledge EEwwards et al. (2022) Pei et al. (2023): ) Liu et al. (2023b). aiming boosting the performance of downstream applications such as property prediction and molecular
design. A critical issue in these model designs lies in how to efficiently mine molecular-related
knowledge from SMILES data. This paper address this issue by presenting SMILES LM with enhanced modeling capabilities.
"Corresponding Authors.
'Code is released http:///gttlllcoo///noggaaiigimt

---

Published as a conference paper at ICLR 2025
Current SMILES LMs often adopt strategies used in natural language processing, such predicting missing tokens corrupted SMILES sequences (e.g., Masked Language Modeling (MLM) and Causal Language Modeling (CLM)). However, this approach introduce several challenges:
ii Unlike natural language, where individual tokens represent independent semantic units like subwords, words or phrases, SMILES data tokens correspond to single atoms, chemical bonds, or special symbols. Molecular functionality, however, is more closely tied to specific substructures (c.g., functional groups). The SMILES LMs focusing only individual tokens in the SMILES context may fail to capture the semantic information of these substructures.
(ii) Predicting single missing token in given SMILES context is often trivial, leading to rapid saturation of model capacity during training. This limits the model's s ability to acquire more nuanced molecular knowledge, and affects its scalability and its effectiveness in generalizing to diverse molecular data.
(iii) Since these models are trained on corrupted SMILES sequences containing special symbols like [MASK] which do not appear in real-world SMILES data they face challenges in modeling the semantic content of complete SMILES sequences.
To address these challenges, we propose an edit-based SMILES language model with fragment-level supervision:
(i) To enable the model to learn richer substructure-relatee molecular information, we intro- duce fragment-level supervision. By randomly removing substructures from molecules and train the model to recover the missing information, we encourage it to acquire more comprehensive fragment-level semantic knowledge.
iii) We design an edit-based pre-training objective, allowing the model to process valid SMILES sequences and to restore missing substructures through an editing process.
In summary, our contributions are threefold:
We analyze the behavior of current SMILES masked language models (MLMs) during pre-training phase and downstream tasks, identifying their rapid saturation problem in pre-training and their limited ability to capture the molccular substructure information. Previous studies lack systematic exploration of these issues.
We introduce the first edit-based pre-trained LM for SMILES, which transforms valid SMILES sequences into structurally related variants. This approach resolves the train-inference mismatch issue in existing SMILES LMs. The integration of fragment-leve supervision further enhances our model's ability to learn richer semantic information from SMILES and improves its overall performance.
Extensive experiments demonstrate that SMI-EDITOR achieves state-of-the-art performance across multiple molecular property prediction tasks, surpassing several 3D molecular representation models. Ablation studies confirm the effectiveness and scalability of SMI-EDITOR.
2 UNDERSTANDING THE BEHAVIOR OF MLM
The Masked Language Model (MLM) is a widely used approach for textual information modeling and has been extensively applied to SMILES representation learning (Wang et al. 2019a| Chithrananda et al.) 2020 Ross et al., 2022). During the training process, tokens a SMILES sequence representing single atoms, chemical bonds, or special symbols are randomly masked with fixed masked ratio of 15%. The model is then trained to predict these masked tokens accurately, in Figure [ To evaluate the effectiveness and capabilities of MLMs for SMILES data, conducted a comprehensive series of experiments.
[M]
C
M
O
M
4
SMILES MLM
is illustrated
Figure 1: The framework of MLM for SMILES.
2.1 RAPID SATURATION PROBLEM
Rapid Saturation During Pre-training. To investigate whether the SMILES MLM model experi- ences rapid saturation during training and how this issue impacts its scalability, trained MLMs
2

---

Published as a conference paper at ICLR 2025
1.0
8
Train Loss
Email MLM
Been Train Loss
Masked Acturacy
Bosnn ooe
Smail Vardd Losa
Masked Accuracj
Train Loss
NasiRato 25% MUM Train Loss
Naskflato 15% MLM Trai Loss
Masked Accuracy
Big MLM Train Looe
Big MLM Volid Loos
NaskRato 45% MUM Trcin Lose
Small MLM MaskedAcc
Valid
Smaal MLM MaskedAoc
Naskllato M8M Maxiesfee
Maskwokkc
MasshesAcc
Nassifato 25% MLM Maxledfee
Big MLM Masiedhoc
Big MLM Maskedhoc
Neskffuto455 45% MLM Mazhodfeoe
0.0
20K
40K
100k
100k
Training steps
Training steps
Training steps
(a) Training Curves: Model Scales (b) Validation Curves: Model Scales (c) Training Curves: Mask Ratios
Figure 2: Rapid Saturation Problem. We train SMILES MLMs of various sizes and masking ratios
using the dataset from [hhou et al. (2023). Figure (a) displays the training loss and masking prediction accuracy of different-sized models, showing rapid decrease in loss and an increase accuracy at the start of the training. Figure (b) presents similar trends for the validation set. Figure (c) illustrates the training loss and accuracy for models with different masking ratios, showing similar patterns.
of various scales and compared their training curves (Details of models with different scale can be found in Appendix D). As shown in Figure 2a the training loss decrease rapidly, with the mask- prediction accuracy on the training set exceeding 90% within the first 5,000 steps. By approximately 10,000 steps, all models achieved over 95% accuracy on the training set. A similar rapid saturation phenomenon is observed on the validation set, as shown in Figure 2b] where the validation loss drops sharply after training begins and mask-prediction accuracy rises quickly. The rapid saturation phenomenon is consistent across all models, regardless of scale, including the small model with just 6.7M parameters. These findings indicate that the MLM pre-training task is overly simplistic, enabling even very small models to converge quickly, which restricts the models' capacity and scala- bility for more complex tasks. We also test the performance of MLMs of different sizes and different training steps on downstream tasks (detailed results are in Appendix E The downstream-ttak results further confirm the limited scalability of SMILE ML.ss
Different Mask Ratio Cannot Alleviate Rapid Saturation. One possible reason for the rapid saturation observed in ML. pre-training is the low masking ratio, with only 15% of tokens masked during training. This might provide insufficient training information and make token prediction too easy, leading to rapid saturation. To explore this hypothesis, we trained large-scale MLMs with different mask ratios (i.e.. 15%, 30%, 45%). The training curves, shown in Figure 2c reveal that MLMs with different mask ratios all exhibit a sharp decline in training loss at the beginning of the training process, converging quickly to very low level. Even with mask ratio of 45%. the training loss drops rapidly, and by 10K steps, the mask-prediction accuracy already exceeds 92%. These findings indicate that increasing the mask ratio does not mitigate the rapid saturation problem or enhance the scalability of SMILES MLMs. Instead, the results suggest that the rapid saturation stems not from the masking ratio but from the inherent simplicity of the MLM pre-training task, which fails to provide sufficient complexity or information for modeling SMILES data effectively.
2.2 CHALLENGES IN MODELING SUBSTRUCTURE SEMANTICS
To evaluate the ability of MLMs to learn molecular substructure semantics, such functional groups we design experiments to analyze whether the models can accurately capture functional group information that is closely related to molecular properties. For this purpose, we use two molecular property prediction datasets, ESOL and FreeSolv (Wu et al. 2017), both of which are highly relevant to molecular hydrophilicityy. Specifically, the ESOL dataset provides information on the water solubility of molecules, while the FreeSolv dataset focuses on the hydrogen free energy, both of which are strongly associated with hydrophilic groups within the molecules.
In our approach, we first fine-tune the MLM on these datasets using linear probing. Next, we traverse the SMILES of all molecules in the datasets and remove the hydrophilic groups (e.g., -OH, -COOH, -NH2, etc.) identified in each molecule. We then compare the predicted molecular property values before and after the removal. As a control, we also randomly delete atoms from these molecules and analyzed the predicted changes in molecular properties.
3

---

Published as a conference paper at ICLR 2025
HG Del (Avg=0.14) (Avg: 0.04)
1.2
HG Del (Avg:00.48 499)
Normalized Density
0.8
Normalized Density
(Avg:0.42)
0.6
0.6
0.4
0.4
0.0
Changes in Predicted Values
Changes in Predicted Values
(a) ESOL Datasct
(b) FrecSolv Datasct
Figure 3: Substructure Semantics Modeling. We compared two molecular perturbation meth- removing hydrophilic groups and randomly deleting atoms-and their effects on the model's predictions of hydrophilicity and related properties. Figure (a) presents the impact of these perturba- tions on model predictions in the ESOL dataset, including the distribution of prediction changes. The average prediction change is similar for both methods (40.14 vs. -0.04) and shows similar distributions. Figure (b) shows the effects on the FreeSolv dataset, also with similar average prediction change.
HG Del (Avg:0.19)
HG Del (Avg:0.38) Rand Del (Avg: 0.01)
1.75
Normalized minallzze Density
Rand Del (Avg:0.85)
Normalized Density
0.8
1.25
1.00
0.75
0.50
0.2
0.25
1.5
-1.0
-0.5
0.0
0.5
1.0
1.5
2.0
0.00
0.75
-0.50
-B25
0.00 0.25 0.50 0.75 1.00 1.55
Changes Predicted Values
Changes in Predicted Values
(a) ESOL Dataset
(b) FreeSolv Dataset
Figure 4: Substructure Semantics Modeling from SMI-EDITOR. We compared the effects of two molecular perturbation methods on the SMI-EDITOR' 's predictions of hydrophilicity and related properties. Figure (a) and Figure (b) show that the impact of deleting hydrophilic groups (HG Del) and randomly deleting atoms (Rand Del) on the model's predictions differs significantly, both in the average change prediction values and their distributions.
As shown in Figure the changes in predicted values after deleting hydrophilic groups (HG Del) similar to those from random deletions (Rand Del) in both the ESOL and FreeSolv datasets. This indicates that the model struggles to differentiate between the impacts removing hydrophilic groups and that of random atoms on molecular properties. These results further suggest that the MLM fails to effectively capture the semantic information of key molecular substructures in SMILES.
EDIT-BASED PRE-TRAINING FRAMEWORK
To overcome the limitations of MLM- SMILES LMs, we propose novel SMILES LM that employs an edit-based training objective. To enhance the model's ability to capture substructure semantics, we introduce fragment-level supervision during pre-training. This involves randomly discarding substructures and requiring the model predict the missing components. This method enables the model to effectively learn substructure semantics. MLMs only operate on corrupted SMILES contexts with unreal [MASK] symbols, leading to inconsistencies between training and testing. In contrast, our approach inputs complete and valid SMILES sequences into the model, seeking to reconstruct the missing substructures through an editing- approach. Moreover, the editing framework offers greater flexibility compared to MLM,, imposes specific restrictions
4

---

Published as a conference paper at ICLR 2025
Data Processing
Original
Expert Editing Policy
Fragmentation
SMILES Encoder
Token Prediction
Supervision Signals
Transformer Block
Fragments Dropping
Expert Input
Transformer Block
Placeholder Insertion
OH
Insertion
Transformer Block
0c1ecore1C/=000
Resulting SMILES
Embedding Layer
col-09Ocleccc1((-0000
Taken Deletion
Figure 5: Overall Framework of SMI-EDITOR. The framework includes data processing module, a SMILES encoder, and an edit-based pre-training process. In data processing, some fragments of the input molecule are randomly removed, and the resulting SMILES is fed into the model. The pre-training goal for the model to edit the corrupted SMILES to recover the original SMILES. To enable this, we add three different heads for token deletion, placeholder insertion, and token prediction to the SMILES encoder (see Appendix ( for details). An expert provides training signals for these operations to help the model learn how recover the original SMILES through editing.
on input forms. This flexibility allows us to create more versatile model inputs by removing certain substructures from molecule, converting it back SMILES, and then feeding it into the model. In this section, we provide an in-depth discussion SMI-EDITOR, focusing on its model design and its pre-training framework.
3.1 SMILES ENCODER WITH EDITING OPERATIONS
In the edit-based pre-training process, the model learn to model the editing operations. Specifically, given SMILES sequence with missing substructures, the model needs accurately predict the editing operations required to reconstruct these missing substructures. To achieve this, we designed SMILES encoder capable of supporting editing-operation modeling.
Model Architecture. The architecture of the model is Transformer Encoder composed of multiple stacked Transformer blocks. Each transformer block contains multi-head self atff-attention layer and weed-forward layer (Vaswani et al. 2017). The SMILES representations extracted by the Transformer Encoder are then passed to the Editing Operations Head, which predicts the necessary editing operations. Similar existing Edit-based LMs (Gu et al. 2019). the model handles types of editing operations: deletion and insertion. Specifically, it completes missing parts of a SMILES sequence by removing redundant parts and inserting missing substructures. To support these tasks, the architecture includes dedicated prediction heads for both editing operations, enabling the model effectively model editing tasks.
Deletion Operations Head. For a given input token, deletion is binary classification problem deciding whether to delete or retain the token. Let af denote representation of the i-th input token extracted by the Encoder. The probability deleting the i-th token, denoted as xde (i), can be expressed as:
(1) Softmax 2
where we have the weight matrix Wa of size x 2, and H is the hidden size.
Insertion Operations Heads. Modeling insertion operations is more complex than deletion. In- spired by LevT Gu et al. (2019), the insertion operation split into two steps. In the first step. the model predicts the positions and number of tokens insert into the original input sequence.
5

---

Published as a conference paper at ICLR 2025
Placeholders [P] then inserted at these positions, representing the locations where tokens will be added. Later on, in the second step, the model predicts the actual tokens for each placeholder [P].
Given the ordinary length of SMILES sequences, we constrain the model to insert at most 255 tokens time. Therefore, this step is framed as 256-class classification problem for each token position. The probability of inserting tokens at the i-th position, denoted as The (t), is defined as:
b (i) Softmax (
where Win is matrix of size H x 256.
In the second step, for each placeholder symbol [P], the model predicts the token to insert. This step resembles MLM pre-training tasks, where the model predicts the probability distribution over the vocabulary for the token at each placeholder. The probability distribution for the token at the i-th position, denoted as mokk (i), is defined as:
tok (i) = Softmax WWT 25 n
tok
where boo is a matrix of size H x vob, where vob represents the size of the vocabulary.
3.2 EDIT-BASED PRE-TRAINE TRE-TRAINING WITH FRAGMENT-LEVEL SUPERVISION
After constructing the SMILES encoder with editing operations, the next crucial step is to develop an edit-based pre-training framework that incorporates fragment-level self-supervised training signals. Unlike traditional MLMs, our edit-based model transforms a valid SMILES input into the target SMILES by modeling the necessary editing operations. This process begins by fragmenting the input molecule using rule-based molecule fragmentation, and then randomly selected some fragments to discard from the original molecule. The resulting corrupted molecule is reconstructed into SMILES representation and fed into the SMILES Encoder.
Molecule Fragmentation and Fragments Dropping. To provide the model with fragment-level training signals, we first split the input molecule M into multiple fragments {f, ..... A popular method for molecular fragmentation is the BRICS algorithm (Degen et al. 2008), which divides molecule into fragments based on predefined rules such as functional groups. However, BRICS often generates relatively large fragments, and removing these fragments can overly disrupt the molecule, potentially eliminating critical structures like rings. To address this, we adopt modified fragmentation approach on RMCF (Wang et al., (022a), which further splits connections between rings and side chains on top of BRICS, resulting in smaller molecular fragments. After cutting the molecule, we randomly select and discard subset of the fragments with predefined probability. The remaining fragments then reassembled into corrupted molecule M. The model is tasked with reconstructing the original molecule's SMILES M from the given corrupted SMILES M through an edit-based approach.
based Training Objective with Dual Loss. The core pre-training task of the SMI-EDITOR model is to take the SMILES of a corrupted molecule M as input and attempt to reconstruct the SMILES of the original molecule M through an editing process, by accurately modeling deletion and insertion operations. However, traditional edit-based models like LevT only provide insufficient training signals for deletion. It only teaches the model remove incorrect tokens it inserted. This constraint hinders the model's 's ability to effectively learn how to delete the incorrect parts in the input SMILES, To overcome this problem, we introduce a dual deletion loss, which supervises the model to correctly remove erroneous tokens from the corrupted molecule input M.
We adopt the imitation imitation-learnnng method from LevT, which supervises the model to minimize the Levenshtein distance between the input and target output. The dual deletion objective defined as:
CD
DDullDel_
Σ
log del (d
d7ea

---

Published as a conference paper at ICLR 2025
where d' is the optimal deletion action determined by expert to minimize the Levenshtein distance to the target output y which is the SMILES of molecule M. This is formulated as d' = argmingD(y' e(M,, d)), where D is the Levenshtein distance, m is the Deletion Classifier, and represents the environment in LevT's Markov Decision Process. eM,, d) applies the deletion action d to the initial input SMILES M, removing selected tokens.
In addition to the dual deletion loss, we retain LevT's original training objective ClevT (see Appendix B for details). This supervises both deletion and insertion actions by minimizing the Levenshtein distance between the input M and target M. The final training objective for SMI-EDITOR is combination of these two losses:
Lu Cp Dea + CLevT
Through this edit-based pre-training process, we equip the SMI-EDITOR model with fragment-level training signals, enabling it to reconstruct the SMILES a corrupted molecule M into the SMILES of the original molecule M via fragment editing.
4 EXPERIMENTS
In this section, we first evaluate the performance of SMI-EDITOR molecular property prediction tasks and compare with baseline models (see Section 4.2). The results show that SMI-EDITOR outperforms both the MLM and 3D molecular models, achieving cate-of-the-art performance. To further validate the model design and pre-training framework, we conduct ablation studies on training signals and editing operations (see Section 4.3). In addition, analytical experiments confirm that SMI-EDTTOR has stronger ability to capture the semantics molecular substructures compared to MLMs. Analysis of the training curves also demonstrates that SMI-EDITOR mitigates the issue of rapid saturation and enhances the training stability (Section 4.4). We also provide details on the training data, baseline models, and implementation used in the experiments (see Section
4.1 EXPERIMENT SETTINGS
Datasets. For pre-training, which includes SMILES information for 19 million molecules. For fine-tuning, we employ the wicoeyerecognized MoleculeNet benchmark (Wu et al. 2018) (see Appendix [ for more details). We follow the same data split as used by Zhou et al. (203)) and tokenize SMILES sequences with the regular expression from Schwaller al. (2018).
the large-scale molecular dataset provided by Zhou et al. (2023),
Baselines. We evaluate our approach against various supervised learning and pre-training baselines, including both SMILES AMILEE--based and 3D molecular pre-training models. The supervised methods include D-MPNN (Yang et al., (019) and AttentiveFP (Xiong et al. (019), both of which are based on graph neural networks (GNNs). For 2D and 3D molecular pre-training, we consider baseline methods: N-gram (Liu et al. 2019a), GROVER (Rong et al. 2020), GraphMVP (Liu al. 202 I), MolCLR (Wang et al. 2022b), InfoGraph (Sun al. 2019)) Mole-BERT (Xia et al. 2023). 3D InfoMax (Stärk et al. 2022), and MoleculeSDE (Liu al. (2023a). For fair comparison, train SMILES model based on MLM pre-training, referred to as SMI-MLM, using the same training data, model architecture, and training hyperparameters as SMI-EDITOR.
Implementation Details. We use a Transformer block with hidden size of 768 and 12 attention heads, comprising 12 layers in the SMILES encoder, which contains total of 86.3 million trainable parameters. During pre-training, the fragment drop ratio is set to 0.15. For downstream tasks, we use the same fine-tuning dataset established by Uni-Mol. (cf. Appendix G for more details about hyper-parameter configuration.)
4.2 RESULTS ON MOLECULAR PROPERTY CLASSIFICATION TASKS
Tasks Details. We evaluate SMI-EDITOR on the MoleculeNet (Wu et al. 2017) benchmark and compare its performance with baseline models. We evaluate SMI-EDITOR 7 widely-used
7

---

Published as a conference paper at ICLR 2025
Table 1: The overall results on molecule property classification datasets. We report ROC-AUC score (higher is better) under scaffold splitting. The best results are bold. The second-best results are underlined. For more detailed information about the dataset, please refer to Table
Datasets
BACE
BBBP
Tox21f
SIDERT
MUV
ClinToxt
ToxCastf 8575
Meant
Molecules
1531
2039
7831
1427
93087
1478
D-MPNN
80.9
71.0
75.9
$7.0
78.6
90.6
65.5
74.2
Attentive FP
78.4
76.1
76.6
84.7
63.7
72.1
N-GramRF
77.9
69.7
74.3
66.8
76.9
77.5
GROVER
82.6
70.0
74.3
64.8
62.5
81.2
65.4
71.5
GraphMVP
81.2
72.4
75.9
63.9
77.7
79.1
63.1
73.3
InfoGraph
77.8
67.5
73.2
59.9
74.1
76.5
63.7
70.4
MoiCLR
82.4
72.2
75.0
58.9
79.4
91.2
69.2
75.5
Mole-BERT
80.8
71.9
76.8
62.8
78.6
78.9
64.3
73.4
3D InfoMax
79.7
69.1
74.5
60.6
74.4
79.9
64.4
71.8
MoleculeSDE
80.4
73.2
76.5
59.6
79.9
86.6
65.2
74.5
SMI-MLM
77.8
68.6
75.1
61.2
75.1
89.8
64.9
73.2
SMI-EDITOR 80.3
77.4
77.1
63.0
80.2
98.9
67.4
77.8
molecular property prediction tasks ( (see Appendix for details). For all the tasks, take the normalized SMILES information as model input and fine-tuning on each task separately. We use ROC-AUC as the evaluation metric, and the results summarized in Table
Results. SMI-EDITOR achieves state-of-the-art (SOTA) performance on out of tasks, and closely matches the SOTA models the remaining tasks. Compared to the MLM model SMI-MLM, whose training settings, training data, and evaluation processes for downstream tasks are all identical to ours, SMI-EDITOR demonstrates superior performance across all seven tasks. These results validate the effectiveness our pre- treiirannigg framework. Additionally, SMI-EDITOR outperforms several molecular representation learning models that utilize 3D information, indicating that SMILES has the potential of revealing important and rich semantic information related to the spatial molec- ular properties, and that SMI-EDITOR effectively captures such information. SMI-EDITOR also demonstrated the strongest average performance across all tasks, indicating that outperforms other baseline models these prediction tasks overall.
4.3 ABLATION STUDIES
4.3.1
ABLATION STUDIES ON FRAGMENT-LEVEL SUPERVISION
Table 2:
Ablation Studies on Fragment-level Supervision. Fragment-level supervision provide more informative and useful training signals than atom-level supervision and are crucial for helping the model learn multi-level semantic information in molecules.
Method
BACE1
BBBP1
Tox211
SIDER
ToxCast
Mean
Atomipropping
80.0
73.4
59.2
66.6
SMI-EDITOR-
MMM-FDrroR-AtoomMasseee Atskim
80.4
73.2
75.0
58.3
64.6
70.3
SMI-MLM
77.8
68.6
75.1
61.2
64.9
69.5
SMI-EDITOR
80.3
77.4
77.1
63.0
67.4
73.0
Experimental Settings. To explore the impact of fragment-level supervision signals versus the atom-level signals on model performance, we additionally train SMI-EDITOR's variants using two different pre-training strategies. The first model, S.M-EEDrroR-AoomDDpppiiii replaces the fragment dropping process in pre-training with random atom dropping. After discarding certain atoms, input the modified SMILES into the model, asking it to restore the original SMILES through an editing approach. The second model, SMI- E AtomsMasking, uses random token masking similar to MIM, where selected tokens replaced with [MASK], and the model is tasked
8

---

Published as a conference paper at ICLR 2025
with restoring the original SMILES via editing. The performance of these models is presented in Table 2
Results Analysis. The results show a significant decline performance when fragment-dropping is replaced with mandom-ttom-droppee that the fragment-level supervision signal enables the model to learn more important and nuanced semantic information. Furthermore, when random atom dropping is replaced with random token mask- ing, performance decreases again (SMI-EDITOR- SAATITORRR-AtMMaASSS AtomsMasking vs. This suggests that while both sadddo--atom-maskiin and manoomatoom-rropiin introduce atom-level training signals, the introduction of the unrealistic special symbol [MASA] through token masking adversely affects model performance. Compared these two models, SMI-MLM exhibits even poorer performance, demonstrating that this editing training approach effectively helps the model learn richer semantic knowledge.
vs. SMI-EDITOR), indicating
EEDTOOR-AtossDrog
4.3.2 ABLATION STUDIES ON EDITING OPERATIONS
Table 3:
Ablation Studies on Editing Operations. The placeholder insertion process, which is absent in MIMs, enables the model learn richer and more diverse semantic information.
Method
BACE!
BBBP1
Tox211
SIDER1
ToxCast
Mean
w/o Plhins
76.1
69.7
76.9
55.5
66.2
68.9
w/o TokPred
79.8
69.2
75.4
57.4
65.9
69.5
w/o TokDel
79.0
73.5
77.3
61.9
64.9
71.3
w/o DualDel
78.4
70.1
76.4
59.5
64.4
69.8
SMI-EDITOR
80.3
77.4
77.1
63.0
67.4
73.0
Experimental Settings. To investigate the impact of different training signals from the editing operations in the SMI-EDITOR model on its performance, we train four additional variants of the SMI-EDITOR model. These models are obtained by removing the training signals for placeholder insertion, token prediction, token deletion, and dual token deletion (setting the training loss weight to zero), corresponding to the three editing operations in the original LevT model and the dual deletion loss. The detailed results are presented in Table 3
Why SMI-EDITOR is Better than SMI-MLM. The results indicate that the ablation of any of these four editing operations leads to significant decline in model performance. Notably, removing the placeholder insertion operation results in the largest performance drop. This operation primarily models the position of missing fragments within the SMILES, highlighting the importance of teaching the model to predict the locations of these fragments for capturing critical semantic information and improving performance. In contrast, the MLM model attempts to predict masked tokens based on their given positions, which simplifies the pre-training task and limits the model's exposure to important semantic information, ultimately affecting performance. Moreover, SMI-EDITOR provides supervision signals for each token in the sequence, but the MLM model only provides supervision signals for [MASK] tokens, limiting the semantic richness of the model.
Results Analysis:
Dual Deletion Loss is More Useful. The ablation of the dual deletion operation also causes a significant decline in model performance, with a more pronounced drop than when token deletion removed. This indicates that the dualldeletionnoojeettv of SMI-EDITOR provides more useful and richer training signals than token-deletion objective in LevT.
Results Analysis:
4.4 ANALYTICAL EXPERIMENTS
SMI-EDITOR Understands Substructure Semanties. Similar to the analysis in Section 2.2 tested SMI-EDITOR'S response to two different molecular perturbation methods on the ESOL and FreeSolv datasets. As shown in Figure 4 compared it to the results in Figure 3 SMI-EDITOR exhibits distinct prediction changes for the two perturbation methods on both the ESOL and FreeSoly datasets. This indicates that SMI-EDITOR can clearly differentiate between the impact of removing

---

Published as a conference paper at ICLR 2025
10
12
Train Loss
Valid Loss
Small Model Train Loss
Small Model Valid Loss
Base Model Train Loss
Base Model Valid Loss
Big Model Train Loss
Big Model Valid Loss
20K
40K
60K
80K
100k
0
20K
40K
60K
80K
100k
Training steps
Training steps
(a) Training Curves
(h) Validation Curves
Figure 6: The training and validation loss curves of different-sized SMI-EDITOR models, The loss curves consistently show stable downward trend throughout the training process, and the model loss gradually decreases as the model size increases.
hydrophilic groups and randomly-selected atoms on molecular properties, demonstrating that it models the semantics of key molecular substructures more effectively than the MLM.
SMI-EDITOR Enhances Training Stability and Model Scalability. We train SMI-EDITOR of different sizes and compare their training curve variations. As shown in Figure 6 the losses of SMI-EDITOR models consistently exhibit a more pronounced downward trend throughout the training process compared to the MLMs (see Figure 2a). further alleviating the rapid saturation problem. Additionally, unlike the MLM, the training loss of the SMI-EDITOR shows more distinct differences across sizes. As the model size increases, the loss steadily decreases, with the larger model (Big Model) converging more stably than the MIM, indicating better scalability for SMI-EDITOR. We also analyze the training and validation loss curves for the three types of editing operations in SMI-EDITOR, confirming the model's 's scalability during pre-training (see Appendix Additionally, evaluate the performance of SMI-EDITOR models of different sizes downstream tasks, demonstrating that SMI-EDITOR exhibits better scalability and stability compared to the MLM (i.e., SMI-MLM) (see Appendix details).
(ornndix([oor [[ror details).
5 CONCLUSIONS
In this paper, we analyze the behavior and shortcomings of masked language models (MLMs) on SMILES data. By examining the training curves, we demonstrate that training MLMs SMILES data encounters rapid saturation issues. Further analytical experiments reveal that MLMs struggle to effectively capture the semantics of important molecular substructures. To address these issues, we propose the edit-based pre-training molecular representation learning framework SMI-EDITOR, which is specifically designed to capture substructure semantics by learning how to recover the missing fragments through edit operations. Extensive experiments on molecular property prediction tasks validate the effectiveness of SMI-EDITOR and ablation studies confirm the advantages of its design over traditional ML.S in modeling molecular substructure semantics and training stability.
ACKNOWLEDGEMEENT
We would like to thank Kuangqi Zhou from Bytedance for his insightful discussions on the project. We also thank other members from Dlib in Peking University for their valuable feedback given during the internal discussions. This paper is partially supported by National Natural Science Foundation of China with Grant Number 62276002.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Viraj Bagal, Rishal Aggarwal, PK Vinod, and U Deva Priyakumar. Molgpt: molecular generation using a transformerrdecoder model. Journal of Chemical Information and Modeling, 62(9): 2064-2076, 2021.
Seyone Chithrananda, Gabriel Grand, and Bharath Ramsundar. Chemberta: large-scale self- supervised pretraining for molecular property prediction. arXiv preprint arXiv:2010.09885, 2020.
Jorg Degen, Christof Beegshhidd-Gerlache Andrea Zaliani, and Matthias Rarey. On the art of compiling and using drug- like chemical fragment spaces. ChemMedChem, 3(10):1503, 2008.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXivv1810.04800, 2018.
Carl Edwards, Tuan Lai, Kevin Ros, Garrett Honke, Kyunghyun Cho, and Heng Ji. Translation between molecules and natural language. arXiv preprint arXiv:2204. 11817, 2022.
Bin Feng, Zequn Liu, Nanlan Huang, Zhiping Xiao, Haomiao Zhang, Srbuhi Mirzoyan, Hanwen Xu, Jiaran Hao, Yinghui Xu, Ming Zhang. et al. bioactivity foundation model using pairwise meta-learning. Nature Machine Intelligence, 6(8):962-974, 2024.
Zhiyi Fu, Wangchunshu Zhou, Jingjing Xu, Hao Zhou, and Li. Contextual representation learning beyond masked language modeling. arXiv preprint arXiv:2204.04160 2022.
Jiatao Gu, Changhan Wang, and Junbo Zhao. Levenshtein transformer. Advances in neural informa- fion processing systems, 32, 2019.
Yugiang Han, Xiaoyang Xu, Chang- Yu Hsieh, Keyan Ding, Hongxia Xu, Renjun Xu, Tingjun Hou, Qiang Zhang, and Huajun Chen. Retrosynthesis prediction with iterative string editing model. Nature Communications, 15(1):4404, 2024.
Thomas Hayes, Roshan Rao, Halil Akin, Nicholas J Sofroniew, Deniz Oktay, Zeming Lin, Robert Verkuil, Vincent Q Tran, Jonathan Deaton, Marius Wiggert, et al. Simulating 500 million years of evolution with a language model. Science, pp. eads0018, 2025.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy. Spanbert: Improving pre-training by representing and predicting spans. Transactions of the association for computational linguistics, 8:64-77, 2020.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint MxXiv:141..6980 2014.
Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, al. Evolutionary-scale prediction of atomic-level protein structure with a language model. Science, 379(6637):1123-111 2023.
Shengchao Liu, Mehmet F Demirel, and Yingyu Liang. N-gram graph: Simple unsupervised repre- sentation for graphs, with applications to molecules. Advances in neural information processing systems, 32, 2019a.
Shengchao Liu, Hanchen Wang, Weiyang Liu, Joan Lasenby, Hongyu Guo, and Jian Tang. Pre- training molecular graph representation with 3d geometry. arXiv preprint 10.0722 2021.
Shengehao Liu, Weitao Du, Zhi-Ming Ma, Hongyu Guo, and Jian Tang. A group symmetric stochastic differential equation model for molecule multi-modal pretraining. In International Conference Machine Learning, pp. 21497-21526. PMLR, 2023a.
Yinhan Liu. Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1907. 1692, 2019b.

---

Published as a conference paper at ICLR 2025
Zequn Liu, Wei Zhang, Yingee Xia, Lijun Wu, Shufang Xie, Tao Qin, Ming Zhang, and Tie- Yan Liu. Molxpt: Wrapping molecules with text for generative pre-training. arXiv preprint arXiv:2305.10688 2023b.
Jonathan Mallinson, Aliaksei Severyn, Eric Malmi, and Guillermo Garrido. Felix: Flexible text editing through tagging and insertion. arXiv preprint arXiv:2003. 2020.
Eric Malmi, Sebastian Krause, Sascha Rothe, Daniil Mirylenka, and Aliaksei Severyn. Encode, tag, realize: High-preeision text editing, arXiv preprint arXiv:I909.00 187, 2019.
Qizhi Pei, Wei Zhang, Jinhua Zhu, Kehan Wu, Kaiyuan Gao, Lijun Wu, Yingee Xia, and Rui Yan. Biot5: Enriching cross-modal integration in biology with chemical knowledge and natural language associations. arXiv preprini arXiv:23 2023.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1 2020.
Yu Rong, Yatao Bian, Tingyang Xu, Weiyang Xie, Ying Wei, Wenbing Huang, and Junzhou Huang. Self-supervised graph transformer on large-scale molecular data. Advances Neural Information Processing Systems, 11112559-11551, 2020.
Jerret Ross, Brian Belgodere, Vijil Chenthamarakshan, Inkit Padhi, Youssef Mroueh, and Payel Das. Large-scale chemical language representations capture molecular structure and properties. Nature Machine Intelligence, (212):1256-1220 2022.
Philippe Schwaller, Theophile Gaudin, David Lanyi, Costas Bekas, and Teodoro Laino. "found in translation": predicting outcomes of complex organic chemistry reactions using neural sequence- to-sequence models. Chemical science, (228):601160999 2018.
Philippe Schwaller et al. Molecular transformer: a model for uncertaintyy-cliiratee chemical reaction prediction, ACS central science, 5(9):1572-1583 2019.
Hannes Stärk, Dominique Beaini, Gabriele Corso, Prudencio Tossou, Christian Dallago, Stephan Günnemann, and Pietro Liò. 3d infomax improves gnns for molecular property prediction. In International Conference on Machine Learning, pp. 20502. PMLR, 2022.
Fan-Yun Sun, Jordan Hoffmann, Vikas Verma, and Jian Tang. Infograph: Unsupervised and semi- supervised graph-level representation learning via mutual information maximization. arXiv preprint arXiv:1908. 01000, 2019.
Raymond Hendy Susanto, Shamil Chollampatt, and Liling Tan. Lexically constrained neural machine translation with levenshtein transformer. arXiv preprint arXiv:2004. 12681, 2020.
Xiaochu Tong, Xiaohong Liu, Xiaoqin Tan, Xutong Li, Jiaxin Jiang, Zhaoping Xiong, Tingyang Xu, Hualiang Jiang, Nan Qiao, and Mingyue Zheng. Generative models for de novo drug design. Journal of Medicinal Chemistry, 64(19):140) 1-14027, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Iliia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017
Lihao Wang, Yi Zhou, Yiqun Wang, Xiaoqing Zheng, Xuanjing Huang, and Hao Zhou. Regularized molecular conformation fields. Advances Neural Information Processing Systems, 35:18929 18941, 2022a.
Sheng Wang, Yuzhi Guo, Yuhong Wang, Hongmao Sun, and Junzhou Huang. Smiles- bert: large scale unsupervised pre-training for molecular property prediction. In Proceedings of the 10th ACM international conference on bioinformatics, computational biology and health informatics, pp. 429-436, 2019a.
12

---

Published as a conference paper at ICLR 2025
Sheng Wang, Yuzhi Guo, Yuhong Wang. Hongmao Sun, and Junzhou Huang. scale unsupervised pre-training for molecular property prediction. In Xinghua Mindy Shi, Michael Buck, Jian Ma, and Pierangelo Veltri (eds.), Proceedings of the 10th ACM International Conference on Bioinformatics, Computational Biology and Health Informatics, BCB 2019, Niagara Falls, NY, USA, September 7-10, 2019, pp. 429 436. ACM, 1019b. doi: 10. 111114/3337733.3342222 /tt::://ooiiooo org. 114.1 3101////...1111333333333333333 3342188
large
URL
https://doi.oot
Yuyang Wang, Jianren Wang, Zhonglin Cao, and Amir Barati Farimani. Molecular contrastive
learning of representations via graph neural networks. Nature Machine Intelligence, 4(3)279-288 2022b.
Zhenqin Wu, Bharath Ramsundar, Evan N. Feinberg. Joseph Gomes, Caleb Geniesse, Aneesh S. Pappu, Karl Leswing, and Vijay S. Pande. Moleculenet: A benchmark for molecular machine learning. CoRR, abs/1703.00564, 2017. htt.t:t:///arx:: orgg abs/11/3/3 00564
Zhenqin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh Pappu, Karl Leswing. and Vijay Pande. Moleculenet: benchmark for molecular machine learning. Chemical science, 9()):513-553 530, 2018.
Jun Xia, Chengshuai Zhao, Bozhen Hu, Zhangyang Gao, Cheng Tan, Yue Liu, Siyuan Li, and Stan
Li. Mole-bert: Rethinking pre-training graph neural networks for molecules. In The Eleventh International Conference on Learning Representations, 2023.
Zhaoping Xiong. Dingyan Wang. Xiaohong Liu, Feisheng Zhong, Xiaozhe Wan, Xutong Li, Zhaojun Li, Xiaomin Luo, Kaixian Chen, Hualiang Jiang, et al. Pushing the boundaries of molecular representation for drug discovery with the graph attention mechanism. Journal of medicinal chemistry, 63(16):8749-8777 8760,,87000000000 2019.
Junwei Yang, Hanwen Xu, Srbuhi Mirzoyan, Tong Chen, Zixuan Liu, Zequn Liu, Wei Ju, Luchen Liu, Zhiping Xiao, Ming Zhang, et al. Poisoning medical knowledge using large language models. Nature Machine Intelligence, 6(10):1 156-1168, 2024a.
Junwei Yang, Kangjie Zheng, Siyu Long, Zaiqing Nie, Ming Zhang, Xinyu Dai, Wei- Ying Ma, and Hao Zhou. Mol-ae: Auto-encoder based molecular representation learning with 3d cloze test objective. bioRxiv, pp. 2024 04, 2024b.
Kevin Yang, Kyle Swanson, Wengong Jin, Connor Coley, Philipp Eiden, Hua Gao, Angel Guzman- Perez, Timothy Hopper, Brian Kelley, Miriam Mathea, et al. Analyzing learned molecular representations for property prediction. Journal of chemical information and modeling. 59(8): 3370-3388, 2019.
Kangjie Zheng, Longyue Wang, Zhihao Wang, Binqi Chen, Ming Zhang, and Zhaopeng Tu. Towards a unified training for levenshtein transformer. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Kangjie Zheng, Siyu Long, Tianyu Lu, Junwei Yang, Xinyu Dai, Ming Zhang, Zaiqing Nie, Wei-Ying Ma, and Hao Zhou. Esm all-atom: Multi-scale protein language model for unified molecular modeling. In Forty-first International Conference on Machine Learning, 2024.
Gengmo Zhou, Zhifeng Gao, Qiankun Ding, Hang Zheng, Hongteng Xu, Zhewei Wei, Linfeng Zhang, and Guolin Ke. Uni-mol: universal 3d molecular representation learning framework. In The Eleventh International Conference Learning Representations, 2023.
Jinhua Zhu al. Dual-view molecular pre-training. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2023.
13

---

Published as a conference paper at ICLR 2025
A RELATED WORKS
Simplified Molecular Input Line Entry System (SMILES) has emerged as a cornerstone sequential representation for molecular data, making it focal point in molecular representation learning.
A large range of Pre-trained SMILES LMs (Wang et al. 2019a Chithrananda et al. 2020 Ross
al., 2022) have been developed to address challenges SMILES-based molecular modeling,
demonstrating their effectiveness across diverse downstream tasks (Bagal et al., 2021 Tong et al.
202T; Feng et al.) 2024; Yang et al.) 2024a). These models typically leverage techniques like ML.M and autoregressive pre-training to capture the complex syntax and semantics embedded in SMILES sequences. In parallel, edit-based generative models, another important approach to sequence modeling. have gained prominence in broader sequence modeling tasks, including machine translation, text summarization, and grammatical error correction. This section provides a comprehensive overview of these two prominent approaches. We first introduce representative work in Pre-trained SMILES LMs, highlighting their architectures and applications. Then, we discuss edit-based LMs, detailing their methodologies and their relevance to sequence modeling tasks.
A.1 PRE-TRAINED SMILES LANGUAGE MODEL
Similar to text, SMILES sequences encode information in a sequential format. Early pre-trained SMILES LMs adopted methods techniques originally designed for text modeling. [Wang et al.) (2019a) introduced EMILES-BERT, inspired by the BERT model (Devlin et al.) [201]], employing its MLM objective, which has been widely applied various data representation learning tasks (Lin
et al. 2023 Zheng et al., 2024 Yang et al. 2024b; Hayes et al., 2025). Their results demonstrated significant improvements in molecular property prediction tasks. Likewise, Likewise,Chittrranaee Chithrananda et al. (2020) developed ChemBERTa, leveraging the advanced RoBERTa architecture (Liu et al., (019b) to enhance SMILES semantics modeling through MLM pre-training. Further advancing this line of research, Ross et al. (2022) proposed Molformer, trained a larger dataset with MLM training objective, showing that SMILES LMs can effectively capture both molecular properties and structural patterns. These contributions have established MLM--based methods dominant approach in SMILES representation learning. In parallel, generative pre-training approaches have also gained traction in this domain. MolGPT (Bagal al. 202T) employs an autoregressive mechanism, while Tong et al. (2021) applied generative models to drug design tasks. More recently, [Luu et al. (2023b) introduced MolXPT, unifying SMILES and textual data using a generative pre-training strategy. Overall, these works highlight the growing importance of pre-trained SMILES LMs, particularly those based on the MLM objective.
A.2 EDIT-BASED LANGUAGE MODEL
Edit-based sequence generation offers faster and more flexible alternative to traditional autore- gressive methods. Malmi et al. (2019) introduced the LASERTAGGER model, which uses tags (i.e., keep, delete, add) edit sequences. The Felix model (Mallinson al. 2020)) extended this idea by integrating a pointer-based mechanism with an MLM backbone to handle insertion and deletion tasks more efficiently. Recognizing that edit operations from an input sequence
target output can be diverse and difficult to compute directly, Gu et al.) (2019) developed the Levenshtein Transformer (LevT) model, which calculates the minimum levenshtein distance between the input and target sequences. By leveraging this metric, LevT generates an optimal sequence of edit operations, improving performance on tasks such as machine translation and post-editing. LevT was further applied to lexically constrained translation tasks with notable success (Susanto et al.) 2020). To resolve inconsistencies between training and inference, Zheng et al. (2023) introduced a dual training objective, improving the performance of edit-based models in applications such summarization and grammatical error correction. With their efficiency and adaptability, edit- bastd LMs have emerged as highly promising paradigm for sequence modeling across diverse tasks.
B DETAILS OF LEVENSHTEIN TRANSFORMER
The Levenshtein Transformer (LevT) is non-autoregressive edit-based generation model that employs a three-step editing process: token deletion, placeholder insertion, and token prediction. LevT is trained using initation-leaming with a dual policy: (i) learning insert tokens by predicting
14

---

Published as a conference paper at ICLR 2025
those that have been randomly deleted from the complete sequences, and (ii) learning to delete tokens by identifying incorrect tokens generated by an insertion module. Below are more details about the training objective of LevT, denoted as C.axT.
Placeholder Insertion Loss. this step, the model needs to determine how many placeholders [P] should be inserted specific positions in the original input, which will later be replaced by concrete tokens in subsequent steps. This operation is modeled as a classification task that predicts how many words need to be inserted after each token in the input sequence. For practical implementation, LevT limits the maximum number of words that can be inserted after each token to 255. Thus, this step essentially becomes 256-class classification task each token, predicting the number of words (0-255) to insert after each token. This process can be represented as follows:
CB
log T8 (o),
where Da is the optimal placeholder insertion action found by the expert that minimizes the Leven- shtein distance to the target output y" which is the SMILES of molecule M, and can be formalized as .(... Yo is the initial input of the model which is the SMILES of molecule P = argmin,
M D the Levenshtein distance measurement, ndel is LevT's Deletion Classifier, and is the environment in the Markov Decision Process of LevT which receives editing actions and returns the modified sequence, and e(Yo. p) means applies the insertion action p to the initial input sequence Yo (e.g., insert some placeholders in Yo). Details of & can be found in LevT's framework (Gu et al. 2019).
Token Prediction Loss. In this step, the task is to predict an exact token for each placeholder [P] in the sequence Yi = e(Y., p²) that has had placeholders inserted. This process closely resembles that of MLM, it essentially entails classification problem where the number of classes is equal the
size of the vocabulary.
C
Σ 4 >E<
log To0t (411,Y")
where t is the optimal insertion action found by the expert that minimizes the Levenshtein distance to the target output Y*, Y1 is the modified sequence by applying the optimal placeholder action p" to the input sequence Yo. and these terms can be formalized as: 1 argmin, D(y" .YYi.)))
3/1 = =(Yo.p"), d' or p" = argmin,,, :(Yo, {d, p})). T is token classifier.
Token Deletion Loss. the previous insertion step, the model may have inserted incorrect tokens. So in this step. it needs to predict which of the previously-inserted tokens are incorrect and should be deleted. Essentially, this step involves learning how to "correct" the errors made during the insertion phase. Specifically, the input to this step is the output from the insertion module, Y2 e(Y,,t), where represents the actions predicted by the model in the token prediction step. Since the task is to decide whether each token in Y2 should be deleted, this step is essentially binary classification task on each token, represented as:
dde
Σ RLEYS d
log T del
(dG Y2),
where df is the optimal delete action found by the expert that minimizes the Levenshtein distance the target output Y" which is the SMILES of molecule M, and can be formalized dž Drgmin,D(y" =(Y2, -dael is LevT's deletion classifier.
Total Loss. Since the editing process of LevT consists of three steps-toker deletion, placeholder insertion, and token prediction- the overall training objective of Lev' is the sum of the training objectives for these three processes:
CLeeT
Cins + Clok Cdel
In summary, unlike MLMs that provide training signals only for each [MASK] symbol in the input sequence, the LevT model offers training signals for every token in both the Placeholder Insertion and
15

---

Published as a conference paper at ICLR 2025
Token Deletion steps. This design requires the model to determine whether each token in the input sequence should deleted and whether new tokens should be inserted after each existing token, thus providing richer semantic information the model.
C MORE TRAINING CURVES OF SMI-EDITOR
0.90
2.76
0.90
2.50
Train Loss
0.85
TokDel Train Loss [Small
Pinins Trin Loss (Small
Train Loss
TolPPred Train Loss (Smml))
Tok (Bass)
Train
Pihina
TokPree Train Loss (laso)
0.79
coo (8kg
0.70
Pihin
(Big)
(Fig)
0.70
1.50
0.66
1.25
40K
100k
40K Training steps
80K
100k
Training steps
Training steps
(a) Token Deletion
(b) Placeholder Insertion
(c) Token Prediction
Figure 7: Training Loss Curves of Editing Operations. We train SMI-EDITOR models of varying sizes and compare their loss curves during training for three different editing operations. As shown in the results, the loss for the token prediction process represented in Figure (c) is consistently the highest among the three type of losses, while the loss for token deletion is the lowest. Furthermore, as the model size increases, all three types of loss exhibit a stable downward trend.
0.90
2.50
Volid Loss
0.85
TrkDe Valid Loss :5mall
2,25
Pihirs Valid Loss (Smal)
Volid Loss
TooPred Valid Loss (Smal))
0.80
TrADw Valid Lose (Baas)
2,00
Pihirw Weldd Loss (Bass)
Tooeree Velid Loss (Baan)
0.75
Piiirs
Pred Valid
0.68
1.28
0.60
40K
60K
1,00
20K
40K
100k
60K
80K
Training steps
Training steps
Training steps
(a) Token Deletion
(b) Placeholder Insertion
(c) Token Prediction
Figure 8: Validation Loss Curves of Editing Operations. Similar the training loss curves, the validation loss for the token prediction process shown in Figure (c) consistently remains the highest among the three types, while the loss for token deletion the lowest. Additionally, as the model size increases, the validation loss for all three editing operations exhibits stable downward trend.
We present the changes in training and validation loss curves for SMI-EDITOR models of varying sizes during training, As shown in Figure 7 and Figure both training and validation losses for the three types editing operations exhibit a stable downward trend as model scale increases. The loss from the token prediction process consistently constitutes the largest portion of the overall training loss. As is mentioned before, during the edit-based pre-training. the token prediction task similar to that of MI.M, as it involves predicting the real tokens corresponding each placeholder token [P], the token
aiming to restore the complete target SMILES. However, unlike the results in Figure 2 prediction loss in the SMI-EDITOR pre-training does not exhibit a rapid saturation phenomenon in the early stages of training. Even in later training phases, the token prediction loss continues to decline steadily. This highlights the advantage of incorporating fragment-level training signals; by removing substructures and requiring the model to predict them, rather than randomly masking tokens, we establish a training task with improved scalability.
D HYPER-PARAMETER FOR MODELS OF VARYING SCALES
In Table 4 we present the specific training hyperparameters for the models different sizes (i.e., Big, Base, Small) used in this study. Notably, despite the differences in training objectives between
16

---

Published as a conference paper at ICLR 2025
MLM and SMI-EDITOR, all other model settings, including the training hyperparameters (as listed in Table and training datasets, remain consistent to ensure a fair comparison of results.
Table 4:
for pre-train models with different scales.
Model
Max Tokens
Layers
Attn Heads
Embed Dim
FFN Dim
Dropout
Num of Paras
Big
64K
12
768
2048
0.1
50.5M
Base
64K
2048
0.1
19.4M
Small
64K
8
512
1024
0
6.8M
E PERFORMANCE OF MLMs ON DOWNSTREAM TASK
0.76
Test Agg-AUC (Best Valid)
0.74
0.72
0.70
0.66
0.64
Small MLM BBBP Downstream
0.62
Big MLM BBBP Downstream
0.60
20K
40K
60K
80K
100k
Steps
Figure 9: The performance of MLMs of different sizes and training steps BBBP task.
We test the performance of MLMs (i.e., SMI-MLM) of different sizes and different numbers of training steps on the BBBP task. To reduce variability and ensure accuracy, we evaluate each checkpoint the downstream tasks five times and take the mean results. As shown in Figure 9 increasing the model's scale does not consistently improve the model's performance on the downstream tasks. In many cases, small models semantic information captured by larger MLMs does not necessarily translate into better downstream task performance. Instead, larger models exhibit greater variability in their performance compared to small ones. These findings highlight the limited scalability and stability of ML.S.
outperform larger ones. This indicates that the
F PERFORMANCE OF SMI-EDITOR ON DOWNSTREAM TASK
We also test the performance of SMI-EDITOR of different sizes and training steps the BBBP task, and the results are shown in Figure 10 Again, we evaluate each checkpoint on the downstream tasks five times and take the mean result ensure accuracy. Compared the performance of the MLM (Figure 99) the larger SMI-EDITOR model (i.c., Big Model) consistently outperforms the smaller models (i.e., Small Model). As the number of training steps increases, the performance gap between large and small models becomes more pronounced for SMI-EDITOR. In contrast, MLMS do not exhibit this trend, with models of different sizes achieving similar performance on downstream tasks. Moreover, larger MLMs show greater performance fluctuations compared to their smaller counterparts. Notably, the larger SMI-EDITOR model demonstrates superior performance stability, as it does not exhibit increased fluctuations compared to smaller EDITOR models. These results indicate that the SMI-EDE EMI-EDITOR model offers better training stability and model scalability than the MLM.
17

---

Published as a conference paper at ICLR 2025
0.76
Test Agg-AUC (Best Valid)
0.74
0.72
0.70
0.66
0.64
Small SMI-Editor BBBP Downstream
0.62
Big SMI-Editor BBBP Downstream
20K
40K
60K
80K
100k
Steps
Figure 10: The performance of SMI-EDITOR of different sizes and training steps BBBP task.
G HYPER-PARAMETER CONFIGURATION FOR PRE-TRAINING
We implement SMI-EDITOR using 12 stacked Transformer layers, each with 12 attention heads. The model dimension and feedforward dimension each Transformer layer are 768 and 3,072, respectively. The total number of SMI-EDITOR' 's parameters achieves 86.3M. We use Adam (Kingma Ba 2014) optimizer and polynomial learning rate scheduler to train SMI-EDITOR, and set the learning rate as 5e - 4 and warmup step 10K. The total training step is 120K and each batch has 64k tokens at maximum. We implement the SMI-EDITOR model using the Fairseq library and train SMI-EDITOR on four RTX3090 GPUs for about day.
For more pre-training ppper-parameters please refer to Table
Table 5: SMI-EDITOR hyper-parameters for pre-training.
paprr-parameters
Value
Learning polynomial decay
Sc-4
LR scheduler
Warmup updates
10K
Max updates
120K
Max tokens
64k
FFN dropout
0.1
Attention dropout
Activation dropout
Num of layers
12
Num of attention heads
12
Encoder embedding dim
768
Encoder FFN dim
3072
Adam (3.. 82)
(0.9,0.98)
Fragments Drop ratio
0.15
Vocabulary size
369
Activation function
GELU
Weight Decay
0.0
Clip Norm
1.0
httts::/faarrre..eaaaeeeee readtheocs. wttp://oo.....ine/////aaaaaaaaae
18

---

Published as a conference paper at ICLR 2025
H HYPER-PARAMETER CONFIGURATION FOR FINB-TUNING
In different downstream task, different syper--arameters. For detailed fine-tuning hyper- parameters, please refer to 6
Table 6: SMI-EDITOR paremtparaeeters for fine-tuning.
Tasks
Epochs
Batch size
Learning rate
Warmup Ratio
Dropout
Pooler-dropout
BACE
60
64
le-4
0.06
0.1
0.2
BBBP
40
128
4e-4
0.06
0.1
0.1
TOX21
80
128
le-4
0.06
0.1
0.1
SIDER
100
5e-4
MUV
40
128
2e-5
0.2
0.1
0.1
ClinTox
100
256
5c-5
0.1
0.1
0.5
ToxCast
80
64
lc-4
0.06
0.1
0.1
I DETAILS OF FINE-TUNING DATASETS
We perform comprehensive set of experiments on the MoleculeNet(Wu Wu et al. 2018) benchmark. focusing on the molecular property prediction task. MoleculeNet has emerged as one of the most widely recognized and utilized benchmarks in the field of molecular property prediction, providing standardized platform for evaluating machine learning models' performances on evaluating molecular properties. Its datasets encompass broad range of molecular tasks, and address diverse and practical scientific problems such as drug discovery, toxicity prediction and so on.
In this section, provide detailed summary of the statistics and fundamental characteristics of the MoleculeNet datasets we use in Ta This table offers information about the dataset sizes, task types, and compositions, providing readers with essential background information better understand the experimental setup and subsequent analysis.
Table 7: Summary information of the MoleculeNet benchmark datasets.
Dataset
Tasks
Task type
Molecules (train/valid/test)
Describe
ESOL FreeSolv
1
Regression Regression Regression
902/1 13
Water solubility
513/64/64
Hydrogen free energy
Lipo
3,360/420/420
Octanol/water distribu- tion ratio, coefficient Binding results of human BACE-1 inhibitors Blood-brain barrier pene- tration
BACE
Classification
1,210/151/151
BBBP
Classification
1,631/204/204
ClinTox
2
Multi-label classification
1,182/148/148
Clinical trial toxicity and
FDA approval status Qualitative toxicity mea-
Tox21
12
Multi-label classification
6,264/783/783
surements
ToxCast
617
Multi-labal classification
6,860/858/858
Toxicology data based on in vitro screening Adverse drug reactions to the 27 systemic organs subset of PubChem BioAssay
SIDER
27
Multi-label classification
1,141/143/143
MUV
17
Multi-label classification
74.46999.309/9.309
19

---

Published as a conference paper at ICLR 2025
J PERFORMANCE OF SMI-EDITOR ON DEEPCHEM DATA
We evaluated the performance of SMI-EDITOR (pre-trained on datasets provided by Ross et al. (2022) on various downstream tasks of MoleculeNet benchmark using the data splits provided by DeepChem In our previous experiments, our results were based on different data split, which made it less convincing to compare our model against others built on this dataset. Therefore, we re-tested SMI-EDITOR on DeepChem splits and included comparisons with more baseline models. Detailed results are presented in Table g As shown in Table 8 EMI-EDITOR achieves significant performance gains over baseline models, reaching dfate-of-the-ar levels with noticeable mean performance improvements. Below a detailed analysis of these results:
SMI-EDITOR outperforms models trained with various paradigms: Measured by the mean performance, SMI-EDITOR surpasses molecular representation learning models like Mol- CLR and DMPTF, which use contrastive pretraining, well as models like ChemBerta and SMI-MLM, which use masked language modeling. also outperforms autoregressive LMs like Galactica and graph-base baspdd models like MoICLR, MGSSL, and MoMu. These results highlight the potential of SMILES LMs.
SMI-EDITOR achieves competitive performance with less training data: SMI-EDITOR outperforms DMPrr, which is trained over 100 million compounds, despite using only 19 million compounds for training. This demonstrates SMI-EDITOR' 's higher data efficiency, enabled by its ability to effectively leverage substructure information from SMILES sequences.
Table 8: Mean results on MoleculeNet datasets using DeepChem splits. ROC-AUC scores (higher is better) are reported for all tasks. The best results are bolded
Method
BBBP
Tox211
ClinToxf
HIVI
BACE1
SIDERT
Mean
GEM
72.4(0.4)
70.1
90.1(1.3)
80.6(0.9)
85.6(1.1)
67.2(0.4)
79.0
ChemBerta
64.3
90.6
62.2
MolCLR
73.6(0.5)
79.8(0.7)
93.2(1.7)
80.6(1.1)
89.0(0.3)
68.0(1.1)
80.7
MGSSL
70.5(1.1)
76.5(0.4)
80.7(2.2)
79.5(1.1)
79.7(0.8)
61.8(0.7)
74.8
DMPTE
78.1(0.5)
78.8(0.5)
95.0(0.5)
81.0(0.7)
89.3(0.9)
69.2(0.7)
81.9
MoMu
70.5(2.0)
75.6(0.3)
79.9(4.1)
76.2(0.9)
77.1(1.)
60.5(0.9)
73.3
MMI
89.4(1.9)
76.2(1.6)
90.6(1.8)
79.8(1.2)
86.6(0.4)
66.5(0.5)
81.5
-MIEEDITOR
93.5(2.2)
81.4(1.1)
95.2(1.3)
81.6(0.7)
89.9(0.2)
69.8(0.6)
85.2
K PERFORMANCE ADVANTAGES SMI-EDITOR OVER AUTO-REGRESSIVE MODELS
To comprehensively compare SMI-EDITTO with autoregressive models, we trained a decoder-only model with identical architecture and size to SMI-EDITOR using an autoregressive eroguage-moeeling objective, referred to as SMI-GPT. We evaluated SMI-GPT's performance across several downstream tasks, with results shown in Table The findings indicate that SMI-EDITOR can perform better than SMI-GPT. Below is an analysis of these results:
K.1 IMPLEMENTATION DETAILS FOR SMI-GPT(NT) AND SMI-GPT(EMB)
This approach uses next-token prediction for downstream classification tasks by appending a special token (e.g., Labelo, Labeli) ) at the end of each SMILES sequence to denote the classes of sample's label. The model learns to predict the correct label token during fine-tuning. (MIIGPT(Emb): The representations of each token in the SMILES string extracted by the SMI-GPT model are processed using mean pooling. The resulting pooled representation is then fed into classification head, which predicts the class the SMILEES.
SMI-GPT(TT):
20

---

Published as a conference paper at ICLR 2025
Table Results of SMI-EDITOR and SMI-GPT on MoleculeNet datasets using DeepChem splits. ROC-AUC scores (higher is better) are reported for all tasks The best results are bolded
Method
BВBP1
Tox211
Clin'Tox
HIV
BACE1
SIDERT
Mean
SMI-GPT(NT)
88.5
74.3
88.9
68.8
76.2
63.7
76.7
SMI-GPT(Emb)
91.2
75.1
91.4
79.4
86.2
67.1
81.7
MoMu
70.5
75.6
79.9
76.2
77.1
60.5
73.3
SMI-MLM
89.4
76.2
90.6
79.8
86.6
66.5
81.5
SMI-EDITOR
93.5
81.4
95.2
81.6
89.9
69.8
85.2
3
1.5
2.8
1.3
24
1.1
2.2
0.9
2
1.8
0.7
1.6
20k
60k
100k
140k
180K
0
40k
80k
120k
160K
(a) Train Loss
(b) Train PPL
Figure 11: The training loss and perplexity (PPL) curves of the SMI-GPT model.
K.2 ADVANTAGES THE ANCODER-ONLL SMI-EDITOR ARCHITECTURE
As shown in Table 9 SMI-EDITOR consistently outperforms SMI-GPT(Emb) and SMI-GPT(NT), highlighting its superior semantic learning capabilities. SMI-GPT(Emb) achieves better performance than SMI-GPT(NT), suggesting that pretraining- based feature transfer preferable for molecular property prediction tasks. Therefore, the encoder-only pre-trained model is highly suitable for molecular property prediction tasks.
K.3 RAPID CONVERGENCE IN AUTOREGRESSIVE LMS
We provide the training curve of the SMI-GPT model in Figure I] which shows that the loss decreases rapidly during the early stages of training. Similarly, the perplexity also drops quickly, reaching approximately 1.6 at the 40K training step. By the end of training, the model's Perplexity falls below 1.6, which is significantly lower than the perplexity typically observed for GPT models trained on text data.
K.4 WHY DOES THIS PHENOMENON OCCUR?
For auto-regressive LMs, each time token generated, it receives all preceding tokens as prefix input. This means that when the model generates tokens at later positions, it has access to more comprehensive contextual information (i.e., longer prefix and more complete sequence information). As result, the prediction difficulty for tokens in later positions is significantly reduced, allowing the model to converge more easily. A key difference between SMI-EDITOR and SMI-GPT that in SMI-EDITOR, each discarded token is predicted independently, with equal importance assigned to the prediction of each token. This enables SMI-EDITOR to more effectively capture the complete semantic information encoded in the tokens.
In summary, compared to LLMs on text data, GPT models on SMILES data converge significantly faster and achieve much lower perplexity. This indicates that SMILES data inherently casier to fit than natural-lann languuge text. Therefore, it is crucial to design effective methods to extract richer seman- tie information from SMILES. Our EDITER represents meaningful and successful exploration
21

---

Published as a conference paper at ICLR 2025
in this direction, highlighting the importance of leveraging substructural fragment information within SMILES data.
L PERFORMANCE OF SMI-EDITOR WITH FRAGMENT CORRECTION
L.1 TRAINING SMI-EDITOR TO CORRECT ERRORS AND REMOVE EXTRANEOUS COMPONENTS DID NOT IMPROVE PERFORMANCE
We implemented a version of SMI-EDITOR that learns to correct erroneous functional groups and remove extraneous substructures, referred to SMI-EDITOR-CH However, AMI-EDITOR -or did not outperform the original EMI-EDITOR on downstream tasks. Table compares the performance of SMI-EDITOR and SMI-EDITOR-CON showing that their performance is similar, demonstrating the limited benefit of incorporating these tasks.
L.2 ANALYSIS OF MII-EDDTOR-COT PERFORMANCE
We attribute SMI-EDITOR-Cor's lack of improvement to the following reasons:
Correcting errors and removing extraneous components provide limited additional training signals: SMI-EDITOR' 's training comprises two major steps: deletion and insertion. During deletion, erroneous functional groups and extraneous substructures are removed, while the insertion step involves learning to recover the correct tokens in the appropriate positions. Thus adding erroneous functional groups or extraneous substructures affects only the deletion step. which is simpler task providing limited information. Morcover, as shown in Table BableBoof of the main text, ablating the token deletion (TokDel) step has minimal performance impact.
Identifying erroneous functional groups and extraneous structures is too simple for the model: SMI OMII-EDITOR-CO constructs erroneous inputs through random substitutions, often resulting in chemically invalid SMILES that are easy for the model identify. Consequently, the simplicity of the training task limits further performance improvement.
Table 10: Performance comparison between SMI-EDITOR-CON and SMI-EDITOR.
Method
BACET
BBBP1
SIDERT
Tox211
ToxCast
Mcant
SMI-EDITOR-Cou
80.6
77.1
62.2
76.8
68.0
72.9
SMI-EDITOR
80.3
77.4
63.0
77.1
67.4
73.0
M HOW THE FRAGMENT DROP RATIO AFFECT SMI-EDITOR
To investigate the impact of the fragment drop ratio on SMI-EDITOR, we trained SMI-EDITOR models with different drop ratios (i.e., 15%. 30%, 45%) and analyzed their training curves and downstream task performance. The results indicate that increasing the drop ratio significantly raises training loss for SMI-EDITOR, indicating that its pretraining task is more challenging than MLM. Below the detailed findings:
M.1 IMPACT ON SMI-EDITOR'S CONVERGENCE
We plotted the training and validation loss curves for SMI-EDITOR with varying drop ratios in Figure 12 The results show that as the drop ratio increases, both training and validation losses rise significantly. Compared to Figure of the paper, the loss increase for pronounced than for MLM, confirming that SMI-EDITOR'S 's task is inherently more challenging.
is more
22

---

Published as a conference paper at ICLR 2025
10
10
Train Loss
Valid Loss
DropRatio 15% Train Loss
7
DropRatio 15% Valid Loss
DropRatio Train
DropRatio 30% Valid
DropRatio 45% Train Loss
DropRatio 45% Valid Loss
5
20K
40K
60K
80K
100k
20K
40K
60K
80K
100k
Training steps
Training steps
(a) Train Loss
(b) Valid Loss
Figure 12: The training loss and valid loss of the SMI-EDITOR with different fragment drop ratios.
M.2 IMPACT ON DOWNSTREAM TASK PERFORMANCE
We evaluated the performance of SMI-EDITOR and MLMs with varying drop mask ratios. The results are summarized Table [I From Table can be observed that as the mask ratio increases, the average performance of the EMI-MLMmode shows significant change, while the performance of the SMI-EDITOR model declines as the drop ratio increases. This indicates that SMI-EDITOR represents a more challenging training task.
Here is a more detailed explanation:
substructures that often serve standalone semantic units. This also makes predicting the discarded fragments more difficult than predicting individual masked tokens. Dropping more substructures severely disrupts the molecular structure, making it harder for the model to reconstruct the original molecule.
SMI-EDITOR discards chemically-meaning
MLM, on the other hand, randomly masks tokens in SMILES sequences. Since SMILES tokens often represent individual atoms or bonds, masking does not typically disrupt the molecular semantics significantly. For instance, masking one or two atoms of functional group like -COOH still leaves enough contextual information to reconstruct it. Additionally, the probability of masking masking mechanism. This explains why MLM performance is less sensitive to mask ratio increases, also reflected in Figure of the paper: Different Mask Ratios Cannot Alleviate Rapid Saturation.
entire functional group low due to MLM's token-based
Table 11:
Performance of SMI-EDITOR and SMI-MLMwith different drop or mask ratios downstream tasks.
Method
BACE1
BBBP
SIDER1
Tox211
ToxCast*
Meant
AMI-MLM(155))
77.8
68.6
61.2
75.1
64.9
69.5
SMI-ML M(30%)
78.3
70.2
58.2
76.0
63.7
69.3
SMI-MLM(45%)
78.4
66.1
59.3
76.4
65.5
69.1
(11-EDITOR(15%)
80.3
77.4
63.0
77.1
67.4
SMI-EDITOR(309/
81.6
73.3
59.6
77.0
66.8
71.7
SMI- (MI-ED)TOR(45%)
79.3
72.2
61.1
77.8
67.1
71.5
23

---

Published as a conference paper at ICLR 2025
Input SMILES: CC(=O))elcc((()ee)
OH
Identifying Bonds to Break
OH
BRICS Bonds
Bonds Between Ring and Side- chain Atoms
Break Bonds
A-OH
Fragments
|+IC(CC
[
:::])ecce((*3))eel
[*:3|0
Drop
Assemble
[':]N[[:2]]
Canonicalize
OH
Santlize
[*33
A-OH
Using dummy atoms to reconstruct topological information
Nclccc(O)eel
Figure 13: An Example Workflow of Molecule Fragmentation and Assemble with Paracetamol
N PERFORMANCE OF SMI-EDITOR ON MOLECULAR PROPERTY REGRESSION TASKS
We evaluated the model's performance on three molecular property regression tasks, as shown in Table SMI-EDITOR achieved the best performance compared baseline models and significantly outperformed the MLM model.
Table 12: Performance of SMI-EDITOR on molecular property regression tasks.
Method
ESOL
FreeSolv]
Lipol
MPNN
0.58
1.150
0.7190
TM
0.700
A-FP
0,503
0.736
0.578
SMI-MLM
0.576
0.709
0.642
SMI-EDITON
0.362
0.524
0.565
O A CASE STUDY FOR FRAGMENTS ASSEMBLE
We provide an example workflow data processing in pre-training with Paracetamol (SMILES: IC(= in Figure 13
24

---

Published as a conference paper at ICLR 2025
P THE SCALABILITY OF SMI-EDITOR
We added results showing the performance of SMI-EDITOR and SMI-MLM models of varying sizes downstream tasks, which further demonstrate SMI-EDITOR's strong scalability. These results are shown in Table [3 is evident that while increasing model size has minimal impact on MLMs, larger SMI-EDITOR models show more consistent performance gains. This confirms the claim that SMI-EDITOR has better scalability compared to MLMs.
Table 13: Performance of SMI-EDITOR and S SMI-MLMwithh different scales on downstream tasks.
Method
BACE1
ВВВР
SIDERT
Tox211
ToxCastf
Meant
SMI-MLM (Smal)
76,8
69.6
60.5
75.3
64.2
69.2
SMI-MI (MI1MM)
76.6
69.3
59.9
75.3
64.4
69.1
(MI-MLM(Big)
77.4
60.8
75.1
65.3
69.4
(ITTEDDTTO((Smaa))
78.3
72.6
59.4
75.6
70.2
SMI-EDITOR
79.2
73.2
61.0
75.7
65.8
71.0
SMI-EDITOR(Big)
79.3
74,2
60.9
76.7
66.4
71.5
Q THE TRAINING COST OF SMI-EDITOR
We measured that the training cost of SMI-EDITOR is approximately three times that of MLMs (i.e.. SMI-MLM) for the same model size, training hyperparameters, and data. However, the training cost of SMI-EDITOR remains acceptable. To better analyze the impact of training cost, we trained an MIM with equivalent computational cost (SMI-MLMM(More)) Results showed that (More) performed worse than the original SMI- ML-MMMMadd significantly lagged behind SMI-EDITOR, highlighting that merely increasing MLM training cost does not yield better results.
Q.1 REASONS FOR HIGHER TRAINING COST SMI-EDITOR
SMI-EDITOR requires computing expert actions (using a computationally expensive dynamic pro- gramming algorithm) and modeling three different editing operations, which introduces additional overhead.
Q.2 ACCEPTABLE TRAINING COST
Training SMI-EDITOR on dataset with 19M compounds using four RTX 3090 GPUs took approxi- mately 24.6 hours. Scaling SMI-EDITOR to larger datasets (e.g., more than 100M compounds) is feasible, demonstrating its potential for broader applications.
Q.3 SMI-EDITOR PERFORMS BETTER UNDER THE SAME TRAINING COST WITH MILM
We trained SMI- AMI-MLM(Mrr)) with the computational cost SMI-EDITOR by inereasing its training steps from 120K to 360K. Table 14 shows that (MMMMMM performs worse than the SMI-EDITOR and original SMI-MLM. This is due rapid saturation issues in MLM training SMILES data. This also indicates that the speed of model training is not the most important factor; what matters more is whether the model can efficiently extract high-quality semantic representations. This highlights the importance of designing more powerful training schemes like SMI-EDITOR effectively extract meaningful information from SMILES.
Q.4 HIGHER PERFORMANCE CEILING FOR SMI-EDITOR
Although the inclusion of Experts slows down the training speed of the SMI-EDITOR model, it also greater scalability and
enriches the semantic information the model learns. This gives higher performance ceiling compared to SMI-MLM. As shown in Table 14 SMI-EDITOR benefits more from more computational resources. This makes AMI-EDITOO a better choice when given sufficient training budget.
25

---

Published as a conference paper at ICLR 2025
Table 14: Performance of SMI-EDITOR and SMI-MLM with training steps on downstream tasks.
Method
BACE!
BBBP
SIDERT
Tox211
ToxCast"
Mean
MMorML
74.3
66.2
49.5
73.3
62.3
SMI-MLM
77.8
68.6
61.2
75.1
64.9
69.5
SMI-EDITOR
80.3
77.4
63.0
77.1
67.4
73.0
R -0.69
Solubility
100
-12
100
200
400
Molecular Weight
Figure 14: The relationship between molecular weight and solubility in the ESOL training set
R A MORE DETAILED ANALYSIS OF THE MODEL'S SUBSTRUCTURE MODELING CAPABILITY
The observed trends for the FreeSolv dataset are fully consistent with our expectations and align with the definition of its physical properties. On the other hand, the performance on the ESOL dataset is influenced by additional factors such as molecular weight. We also designed more analytical experiments further investigate the behavior of the SMI-EDITOR model, and the results demonstrate that the model's behavior aligns with expectations. Detailed explanations are follows.
For the FreeSolv dataset, the observed trends align with its physical property definitions. FreeSolv reflects the hydration free energy of compounds, defined as the free energy change when com- pound transitions from non-dissolved state to a dissolved state. When hydrophilic groups molecule are reduced, the change in hydration free energy increases, leading higher hydration free energy. Therefore, when we remove hydrophilic groups from the molecule, the model predicts an increase in hydration free energy, consistent with the trend observed in Figure 4b which matches expectations.
For the ESOL task, the model predictions significantly influenced by molecular weight The ESOL dataset reflects compound solubility, which is strongly negatively correlated with molecular weight: the larger the molecular weight, the lower the solubility. We plotted scatter diagram (Figure 14) showing the relationship between molecular weight and solubility in the ESOL training set. A clear negative correlation with coefficient of R -0.69 is observed. Consequently, when functional groups or atoms are removed from molecule, its molecular weight decreases, leading the model predict an increase solubility, This explains why, in Figure 4a| the model predicts increased solubility regardless of whether hydrophilic groups or random groups are removed. The increase is more significant with random deletions, demonstrating the model's 's ability to distinguish between hydrophilic group deletions and random deletions.
To eliminate the influence of molecular weight, we designed a hydrophilic group replacement scheme (HG Rep). We replaced all hydrophilic groups in molecule with non-hydrophilic groups of similar molecular weight (e.g., methyl, ethyl, propyl) and compared this hydrophilic group replacement scheme (HG Rep) with random group replacement scheme (Rand Rep), where random groups replaced with others of similar weight. The results, shown in Figure 15 reveal that SMI-EDITOR effectively distinguishes between HG Rep and Rand Rep, demonstrating its ability model key
26

---

Published as a conference paper at ICLR 2025
1.0-
HG Rep (Avg-0.96)
HG Rep (Avg:-0.22 -02)
Normalized Density
Rand Rep (Avig-0, 09)
lormalized Density
0.8-
Rand Rep (Avg: 0.2
0.6-
0.4
0.3
0.2
0.2
0.1
0.0
-2
-1 0
0.0
-2
-1
2
Changes Predicted Values
Changes Predicted Values
(a) SMI-EDITOR
(b) SMI-MLM
Figure 15: Substructure Semantics Modeling on ESOL Dataset. We compared the effects of two molecular perturbation methods the SMI-EDITOR's and SMI-MLM's predictions of hydrophilicity. Figure 1 show that the impact of replacing hydrophilic groups (HG Rep) and randomly replacing atoms (Rand Rep) the model's predictions differs significantly, both in the average change in prediction values and their distributions.
0.6
HG Rep (Avg:-0.36)
Normalized Density
Rand Rep (Avg:-0.17)
0.5
0.4
0.3
0.2
0.1
0.0
Changes in Predicted Values
Figure 16: Substructure Semantics Modeling on ESOL Dataset of Auto-regressive LM.
molecular group semantics. It also correctly predicts that replacing hydrophilic groups reduces molecular solubility.
Furthermore, we plotted the distribution of predicted changes for MLMs and Auto-regressive LMs before and after these replacement operations Figure [6] The results show that these models perform significantly worse than the SMI-EDITOR in distinguishing between random replacements and hydrophilic group replacements. This further highlights the superiority of the SMI-EDITOR in modeling the semantics of molecular substructures.
S SMI-EDITOR' ' PERFORMANCE ON RETROSYNTHESIS PREDICTION TASKS
Considering that the original SMI-EDITOR is an encoder-only model and cannot be directly applied to generative tasks, we further pretrained a model based on encoder-decoder architecture, referred to as SMI-EDDIORR-enn We tested its performance the retrosynthesis prediction task, where achieved state-of-the-arr results. Below is detailed discussion:
SMI-EDITOR-Ges adopts transformer architecture with a base-sized scale (Vaswani et al. 2017). During pretraining, the input to the encoder consists of SMILES strings with missing molecular fragments, while the decoder's pretraining task is to reconstruct the original SMILES. Following approaches commonly used in machine translation
Model Details of SMI-EDITOR-Gon
(Vaswani et al. 2017) the features extracted by the encoder passed to the decoder through
27

---

Published as a conference paper at ICLR 2025
encoder-decoder attention (Vaswani et al. 2017) Compared to SMI-EDITOR, the most significant difference is that the encoder-decoder architecture enables SMI-EDITOR -en to perform sequence- to-sequence generative tasks, allowing to explore the model's capabilities in such tasks.
SMI-EDITOR-Gen Exhibits Strong Performance in Retrosynthesis Prediction Tasks. Following the experimental setup of EditRetro(Han al. 2024). we evaluated SMI-EDITOR-CE retrosynthesis prediction task. During fine-tuning, we applied the same fine-tuning strategies and data augmentation techniques EditRetro. The experimental results, shown in Table [5] demonstrate that SMI-EDITOR achieved strong performance on the USPTO-50K dataset. This validates that the pretraining approach proposed by SMI-E EDITOR also exhibits excellent performance and great potential in generative tasks.
the
Table 15: Top-k exact match accuracy of SMI-EDITOR and baselines on the USPTO-50K dataset.
Top-1 1
Top-3 f
Top-5 1
Top-
RetroPrime
51.4%
70.8%
74.0%
76.1%
Transformer
42.4%
58.6%
63.8%
67.7%
SCROP
43.7%
60.0%
65.2%
68.7%
MEGAN
48.1%
70.7%
78.4%
86.1 %
GTA
51.1%
67.6%
74.8%
81.6%
Retroformer
53.2%
71.1%
76.6%
82.1 %
Graph2Edits
55.1%
77.3%
83.4%
89.4%
R-SMILE
56.3%
79.2%
86.2%
91.0%
EditRetro
60.8%
80.6%
86.0%
90.3%
SMI-EDITOR
61.2%
80.9%
86.4%
89.7%
T SPE TOKENIZER DOES NOT IMPROVE SMILES MLM PERFORMANCE
We trained SMILES MLM with SPE tokenizer, SMI-ML using the architecture and hyperparameters as SMI-EDITOR, and evaluated on multiple tasks. As shown in Table T6 (MS-M performs similarly SMI-MLMand significantly worse than SMI-EDITOR. This demonstrates that introducing SPE cannot replicate the effectiveness of SMI-EDITOR. The reasons follows:
Limited Fragment Diversity: SPE relies fixed vocabulary, limiting the diversity of fragment-level information it can capture. In contrast, SMI-EDITOR dynamically fragments molecules using the BRICS algorithm, capturing wider variety of molecular substructures.
Topology Information Leakage: SPE-based models retain token position information, which is tied to molecular topology in SMILES, making the prediction task easier but less effective.
Lack of Chemical Context: SMI-EDITOR fragments molecules based on chemical rules, allowing to capture substructure information more relevant to molecular properties, unlike SPE, which relies character pair frequencies.
Superior Performance with F Fragment-Level Supervision: A MLM trained with fragment- level supervision, (FFra),M(rrag).. outperforms (MS-ML as shown in Table I6 This validates the effecttivness of SMI-EDITOR'S training approach.
U MASKED SPAN LM DOES NoT IMPROVE SMILES LM PERFORMANCE
To highlight the differences between SMI-EDITOR and Masked Span LMs (MSLMs), we trained SMILES model using MSLM, which randomly masks continuous sequences in SMILES and predicts the missing parts (similar to Sosn et al., (0200). This model, referred to as SMI-MLM(SPANN) shows performance comparable to SMI-MLMbut significantly worse than SMI-EDITOR (see Table [6). This further demonstrates SMI-EDITOR advantages over traditional MSLMs. Reasons for Poor Performance of Traditional MSLMs:
28

---

Published as a conference paper at ICLR 2025
Table 16: Performance comparison of MLMs with different pretraining strategies.
ВACE1
BBBP
SIDER
Tox211
ToxCast
Mean
SMI-MLM
77.8
68.6
61.2
75.1
64.9
69.5
SMI-ML
76.7
71.1
59.3
74.7
65.3
69.4
(MS-ML
78.6
67.2
59.4
76.1
62.3
68.7
SMI-MLM(Frag)
79.4
73.3
62.1
74.0
64.8
70.7
SMI-EDITOR 80.3
77.4
63.0
77.1
67.4
73.0
Differences between Text Data and SMILES Data. Unlike text, molecular data has complex topological structures. In text, adjacent tokens often have strong semantic relevance, and continuous spans convey related information, making span masking effective for learning local semantics. However, SMILES lacks such locality; single functional group may not appear contiguous, and adjacent tokens may lack strong relevance. For example, aromatic rings with multiple substituents often appear discontinuous in SMILES (we provide specific case CASE1). This limits the effectiveness of applying span masking directly to SMILES data.
Traditional MSLM (e.g., TS et al. 2020) and SMI-EDITOP Have Different Implementations: Traditional MSLM is Unsuitable for SMILES Data. Text data's semantic continuity enables models like T5 to use random span masking, where continuous text segments are masked for prediction. In contrast, SMILES lacks this continuity, so S.MIEDDTTR uses fragmentation algorithm to split molecules into chemically meaningful fragments. The model predicts missing fragments, which may correspond continuous SMILES segments. Unlike traditional MSLM, SMI-EDITOR focuses on masking chemically significant fragments, key difference in its design.
Better Performance of SMI-MLM(rrag). The improved performance SMI- MLM(Frag) over EMI-MLM(SPAN, highlights the superiority of SMI-EDITON fragment- level supervision. While (SPAN))(SPAN) uses the traditional MSLM approach, SMI- MLM(Frag) incorporates supervision signals similar to SMI-EDITON enabling it to better capture molecular substructure information.
CASE1 When does SMILES exhibit discontinuity: SMILES linearized representa- tion of graph-structured molecules, which inherently causes discrepancies between molec- ular topology and sequence-level representation. For example, when a ring contains multiple substituents, its representation in SMILES often becomes discontinuous. Con- sider Glibenclamide, COcIcce(Cljc (accccCCcceC(((O)NCCC (
drug used for diabetes treatment, with the canonical SMILES: Here, the bolded atoms originate from the same aromatic ring, but due to the multiple substituents, this ring is represented discontinuously in SMILES. Additionally, the aromatic carbon ce2 adjacent to CCC3 atoms from a distant cycloalkane ring. Such discontinuities are common in SMILES and adversely affect Masked Span LMs.
V COMPARISON BETWEEN SMI-EDITOR AND CONTRASTIVE LEARNING
Similarities: Both contrastive learning and SMI-Editor aim learn alignment.
Contrastive learning aligns representations of different views. The core idea of con- trastive learning is to bring the representations of different views of the same sample (positive pairs) closer while pushing representations of different samples (negative pairs) apart. Essentially, this process learns the correct alignment between views of the same sample.
SMI-Editor aligns representations of missing substructures and contexts. As Fu et al. (0022) noted, MLMs align the representations of contexts and missing words during training. Similarly, SMI-Editor aligns the representations of missing substructures and their contexts. For example, given the input cc the model need to predict the complete molecule
29

---

Published as a conference paper at ICLR 2025
CC(=O)Nelccc()jee! SMI-Editor effectively align the representation of the missing fragment CC(=O) with the context Nelcec(O)cc" through this process.
Differences: The alignment targets differ between the two paradigms.
The representations to be aligned often correspond to different augmented views of the same molecule, such as through atom deletion, bond deletion, or subgraph deletion. These views typically preserve the molecule's overall structure and thus contain global information.
Contrastive learning focuses on global information:
SMI-Editor emphasizes aligning local substructure information with global context: In SMIIEditor, the context typically corresponds to the molecule's backbone, representing global information, while the missing substructures contain local information.
SMI-Editor is more sensitive to local structure information: By aligning local substruc- tures with global context, SMI-Editor learns finer- mner-grained semantics from SMILES data, making it better suited to capturing detailed molecular information than contrastive learning.
W K-FOLD CROSS-VALIDEE
OF THE SMI-EDITOR MODEL.
Using 5-fold setup, we evaluated SIDER, Tox21, ToxCast, Clin'Tox and MUV. The results are shown in Table 17 These results demonstrate that SMI-EDITOR exhibits strong performance and stability across downstream tasks.
's performance on the training sets of BACE, BBBP,
Each dataset was evenly divided into five parts. In each run, one part was selected the validation set, while the remaining four parts were used as the training set. The model was trained and evaluated on validation set. This process was repeated five times to complete all runs.
Table 17: 5-fold cross-validation results of the SMI-EDITOR model.
BACE!
BBBP
SIDER
Tox21
ToxCast
ClinToxt
MUV
Run 1
91.92
97.64
62.59
83.69
75.83
99.76
77.23
Run 2
91.86
96.27
66.89
84.09
73.31
98.73
79.49
Run 3
90.82
98.53
62.60
84.87
73.52
99.6
77.4
Run 4
91.13
98.77
63.32
83.95
74.60
99.82
77.58
Run 5
90.68
97.84
63.50
85.83
75.51
98.61
79.39
Mean
91.28
97.81
63.78
84.48
74.55
99.30
78.23
Std
0.58
0.97
1.78
0.87
1.13
0.59
1.12
X
BROAD APPLICATIONS OF ATOM-LEVEL TOKENIZERS
Currently, many SMILES LMs, including MLM and autoregressive LMs, rely on atom-level tok- enizers to process molecular representations. Atom-level tokenizers break down SMILES strings into individual atomic units or tokens, such as atoms and simple symbols (e.g.. "C", "O", "="). This approach simplifies the tokenization process and aligns well with the intrinsic atomic structure of molecules, enabling models to capture fine-grained atomic interactions and features. For example, MolXPT (Liu et al. (023b) and Dual-view Molecular Pre-training (Zhu et al. 2023 explicitly leverage atvellevel tokenization to enhance the granularity of molecular representations, facilitating downstream tasks such as molecule generation and property prediction.
Atom-level tokenization has the advantage of maintaining straightforward correspondence between the SMILES representation and the underlying molecular structure, making it easier for the model to interpret local chemical environments. This granularity is particularly beneficial for tasks that require precise predictions. For instance, studies such as ChemBERTa (Chithrananda et al. 2020), Molecular Transformer (Schwaller et al.) 2019), and SMILES- BERT (Wang et al. (019b) demonstrate that atom-level tokenization can achieve good performance in molecular property prediction tasks.
30


Reviews:
[{"review_id": "QJWErA8eia", "reviewer": "Reviewer_uWiK", "summary": "This paper identifies three key challenges in existing SMILES masked language models: the neglect of substructural information, overly simplistic training tasks, and a mismatch between training and inference procedures and introduce a SMILES language model employing edit-based, fragment-level supervision to address these challenges.", "questions": "1.\tHow to ensure that left-over fragments can be assembled into a valid molecule? Please provide an exact example.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1.\tThis paper reveals the shortcomings of existing SMILES masked language models through an analysis of experimental results, thereby informing future research directions.\n2.\tThis paper presents a novel SMILES language model that uses edit-based, fragment-level supervision. This approach improves performance on the molecule property prediction task.", "weaknesses": "1.\tEdit-based models, as currently designed, focus solely on restoring removed substructures during pre-training. Other valuable sources of molecular information, such as correcting errors and removing extraneous components, may be neglected.\n2.\tThe data processing and pre-training stages lack clarity. A detailed case study would significantly improve understanding of these processes.\n3.\tThe description of the experimental tasks is inconsistent between the caption of Table 1 and the \" 5.2 RESULTS ON MOLECULAR PROPERTY PREDICTION\" section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "apZVZyZjFc", "reviewer": "Reviewer_mXJn", "summary": "The paper first investigates the problem of MLM training, as train-inference mismatch. To address this, the authors propose the edit-based pre-trained language model, SMI-Editor, for SMILES. As another pre-training strategy, SMI-Editor drops the substructures in a valid SMILES sequence and tries to recover the original molecules.. The extensive experiments demonstrate the effectiveness of SMI-Editor.", "questions": "See weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The model is well-motivated to solve the train-inference mismatch problem in existing MLM paradigm.\n\n2. Extensive experiments indicate the effectiveness of SMI-Editor.\n\n3. The analysis of the rapid saturation problem of MLM and how SMI-Editor can solve this problem is generally convincing.\n\n4. The paper is well-written and easy to follow.", "weaknesses": "1. The authors put a lot efforts to compared SMI-Editor with MLM paradigm. However, there are still other strategies for pre-training the model, for example, contrastive learning. Can the authors also discuss the relations between SMI-Editor and other pre-training paradigms?\n\n2. As the authors state, increasing the mask ratios will influence the convergence in MLM. Would fragment drop ratio also influences the SMI-Editor?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "faWFci14dy", "reviewer": "Reviewer_SiA5", "summary": "This paper proposes an edit-based pre-trained SMILES language model to learn 3D molecular representations and applies the model to several downstream tasks, such as molecular property prediction. The idea of using token edit operations to replace token masking introduces a novel approach with the potential to address the rapid saturation problem. The paper provides comprehensive details on model implementation and includes extensive ablation studies.", "questions": "1. Figure 3: The purpose of this figure is unclear. It would be beneficial to include a comparison with other methods to highlight the advantages of the proposed approach.\n2. Table 1: It is important to specify the amount of data used for fine-tuning to better understand the model's performance.\n3. Generative Tasks: It would be interesting to explore whether this model can be applied to generative tasks, which could broaden its applicability.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Novelty: The use of token edit operations instead of traditional token masking is innovative and could offer significant advantages in terms of model performance and stability.\nComprehensive Implementation: The paper provides detailed descriptions of the model architecture and training procedures.\nExtensive Ablation Studies: The authors have conducted thorough ablation studies to validate the effectiveness of their approach.", "weaknesses": "1. Figure 4: The requirement for an expert to provide training signals is inefficient and limits the size of the training set. This could be a significant drawback in practical applications.\n2. Figure 5: Figures 3 and 5 should be presented together for easier comparison. Additionally, the figure's purpose is not clearly explained.\n3. Figure 6: The figure's ability to demonstrate scalability is questionable. It would be helpful to show whether larger models yield better results to substantiate claims of scalability.\n4. Conclusion: The statement, \"ablation studies confirm the advantages of its design over traditional MLMs in modeling molecular substructure semantics and training stability,\" is not fully supported by the evidence presented in Figure 5. The figure does not convincingly demonstrate that the model understands molecular substructure semantics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gvPNIbdMbK", "reviewer": "Reviewer_Rnmw", "summary": "The paper introduces SMI-Editor, a novel edit-based SMILES language model with fragment-level supervision to improve molecular representation learning. SMI-Editor tries to address the limitations of masked language models (MLMs), such as rapid saturation and limited substructure semantics modeling, by using an LevT-based modeling.", "questions": "- Why not include the performance on FreeSolv and ESOL?\n- What about applying MLM on fragment spans? This could also help alleviate the saturation problem in single-token prediction.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The analysis identifies interesting issues: the MLM model struggles to distinguish between random deletion and hydrophilic deletion, and it quickly saturates on single-token masking.\n- The paper writing is clear and well-illustrated.", "weaknesses": "- LevT isn’t actually an MLM model; it’s a non-autoregressive generative model. From this perspective, the authors' solution for handling corrupted SMILES input is somewhat trivial, as other sequence-based generative models could also address this.\n- There is a discrepancy between the motivation and solution. Neither corrupted SMILES nor fragment-level supervision is directly related to the way LevT models molecules.\n- Limited effectiveness. While I understand the authors' choice of the MoleculeNet baseline without considering powerful graph-based models like Uni-Mol, DVMP_{TF} in Jinhua Zhu et al. is a SMILES-based Transformer encoder and appears to perform much better than the reported results.\n- Figure 5 is confusing; if the model can distinguish hydrophilic groups, wouldn’t performance degrade noticeably after HG deletion?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SRTB0vtLz2", "reviewer": "Reviewer_AGu3", "summary": "In this paper, the authors propose SMILES-Editor, an edit-based pre-trained SMILES language model by introducing a new pre-training strategy that randomly corrupts SMILES strings during the pre-training process and lets the LLM restore the original SMILES strings. Experimental results show that the new proposed pre-training strategy achieves better performance than the previous MLM task in various downstream tasks.", "questions": "1. Could the authors find more evidence to support their claims in Weakness 1?\n2. Could the authors explain more about Weakness 2?\n3. I am wondering about the comparison between different pre-training strategies. Could the authors compare the auto-regressive pre-training strategies in identical decoder-only structures?\n4. What is the time cost or complexity of SMILES-Editor compared to the previous MLM?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The idea of focusing on the atoms and fragments is meaningful and enables better learning of SMILES representations.\n2. The new pre-training task may lead to an improvement in the valid generation of molecule SMILES representations.\n3. The SMILES-Editor shows competitive performance compared to the original MLM, bringing an improvement in the pre-training of SMILES Language Models.", "weaknesses": "1. The authors claim that \"most existing pre-trained SMILES language models (LMs) only provide supervision at the single-token level during pre-training and fail to fully leverage substructural information of molecules\", which lacks enough support. In works like  [1], the tokenizers are initialized with SMILES Pair Encoding. Meanwhile, the sentence-piece method can also find common fragments in molecule SMILES strings, and these common fragments are usually critical functional groups. In this case, I do not agree with this claim.\n2. The novelty of SMILES-Editor is limited. The main contribution of this paper is to propose a new pre-training strategy, while this strategy is not novel, and there is not a significant difference compared to the previous Masked Span Language Modeling used by T5. Furthermore, more advanced LLMs are now using decoder-only structures, while SMILES-Editor can only be adopted in LLMs with encoder structures, which further harms the novelty.\n3. The experiments on the classification tasks do not incorporate the k-fold experiments, which raises concerns about the stability of the method.\n4. The experiment results do not seem satisfying enough. And the comparison should include more pre-training strategies.\n5. In Table 2, the performance of SMI-EDITOR-AtomsMasking in BACE is even better than SMILES-Editor.\n\n#### References\n[1] Li, X., & Fourches, D. (2021). SMILES pair encoding: a data-driven substructure tokenization algorithm for deep learning. Journal of chemical information and modeling, 61(4), 1560-1569.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)