Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Authors propose knowledge distillation from graph Transformer (GT) to SMILES Transformer toward learning a molecular representation with similar performance to Graph-based models while having low inference time (by avoiding smiles-to-graph conversion bottleneck).

Main contributions are:

(1)  An end- to-end SMILES Transformer for molecular representation learning is proposed that is boosted by graph transformer to smiles transformer knowledge distillation.
(2) Biased transformer layers are designed to achieve knowledge transfer; learnable SMILES token embedding are used to encode structural information. 
(3) ST-KD shows competitive results on PCQM4M-LSC and QM9, with fast inference compared to existing graph models. 
(4)
main_review: Strengths:
(1) Knowledge distillation between graph transformer and SMILES transformer is proposed for molecular representation learning. 
(2) Feature distillation and attention transfer losses are used for knowledge transfer.
(3) ST-KD achieves strong performance, similar to graph-based models, on molecular property prediction tasks, with a fast inference time.

Weaknesses:
(1)	Recent SMILES transformer models trained on large data have shown performance similar to state-of-the-art graph-based models without any knowledge distillation, see e.g. Ross et al 2021 https://arxiv.org/abs/2106.09553. Such works should be discussed in comparison to ST-KD.  
(2)	ST-KD performs slightly worse than Graphformer on PCQM4M-LSC as in Table 1. At the same time, the comparisons of ST-KD with Graphformer on QM* and Freesolv are not presented in Table 2. Therefore, it is not evident to what extent knowledge transfer is successful on QM* and Freesolv.
(3)	 It is not clear to what extent ST-KD performance is generalizable beyond the tasks discussed in the paper – for example, Moleculenet classification tasks or on bigger molecules. 
(4)	Graph-to-text knowledge distillation has been investigated before, e.g. https://aclanthology.org/2020.emnlp-main.551.pdf. Such works should be at least cited in this work. 
(5) Inference time comparison is not shown for datasets other than PCQM4M-LS.

summary_of_the_review: The work is interesting, as it explores the idea of knowledge distillation from a Graph transformer to a SMILES transformer for performant molecular prediction with fast inference. However, the idea of cross-modal knowledge distillation is not novel. While it is well understood that the SMILES transformer after KD provides the advantage of fast inference, it is not clear if knowledge distillation is the way to get to SOTA performance with fast inference, as recent pre-trained SMILES-based efficient transformer models are providing competitive performance.  Also, the work does not provide any strong evidence of the  generalizability of the proposed approach.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors provide a knowledge distillation framework to transformer knowledge from graph transformer (teacher) to SMILES transformer (student), where the  SMILES transformer (student) is forced to mimic graph transformer (teacher) 's desired property. 
main_review: The knowledge distillation framework is quite interesting, where the author carefully provide the way to perform it Fig2 and Sec 3.2. 

My major concern about this work is the performance of the SMILES-based transformer is not worse than the graph-based model. In fact, in many cases SMILES-based transformers outperform graph-based models (or at least comparable performances). Why is there a need to perform knowledge distillation to learn a SMILES Transformer (student) from graph Transformer (teacher) ? 


The SMILES-based model (e.g. ST-BASE) used as baseline in this paper seem not a competitive SMILES-based model. Other competitive SMILES based models should be included in the comparison. For example, https://github.com/pschwllr/MolecularTransformer
which has a different way of tokenizing and utilizing SMILES-based input. The SMILES-Transformer's performance can be largely boosted by simple augmentation trick, e,g using random-smiles as augmentation when training the model. The performance of this model may outperform graph-based models. Can you please include the comparison against this work?


Additionally, if we refer to table 2, it seems that there is not a clear improvement of graph-based models over ST-BASE. 

Can the author provide intuition of why the proposed methods ST-KD can outperform both graph and SMILES based methods in table 2, when it is trained to minimize the difference between graph transformer and SMILES transformer (by eq 16)? Should the teacher ( the graph-based methods)  be the upper bond? 


summary_of_the_review: The authors provide an interesting knowledge distillation framework to learn molecule embedding, however, the reviewer is not fully onboard with the motivation of doing the knowledge distillation (e.g. learning SMILES based transformer from graph-based transformer), since smiles-based transformer has a competitive performance.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: High-throughput molecular representation for large chemical databases is one of the kernel requirements for many downstream applications, such as drug discovery and virtual screening etc. Based on the consideration that SMILES strings are the most common storage format for molecules, this paper provides a SMILES-based molecular representation learning model, named ST-KD, by using deep graph models to extract molecular feature from raw SMILES data without SMILES-to-graph conversion. Specifically, the proposed model adopts an end-to-end SMILES Transformer framework for molecular representation learning boosted by Knowledge Distillation. Experimental study shows that ST-KD shows competitive results on latest standard molecular datasets PCQM4M-LSC and QM9, with 3-14 inference speed compared with existing graph models.
main_review: Strengths:
1.	The core ideas of this paper are interesting and well-implemented. To my knowledge, end-to-end SMILES-based molecular fingerprint generators are discussed by many prior works, but none of them have proposed to improve the model with knowledge from existing state-of-the-art graph models. Followed by the key insight, the authors have proposed effective methods to distill knowledge from graph Transformers to SMILES Transformers. 
2.	Convincing experimental results. Results on PCQM4M and ablation studies have corroborated the effectiveness of distillation methods. The model also achieves outstanding performances on downstream tasks like QM9, where the 0.62 multi-mae surpasses all existing models I know by a large margin, including those with molecular conformation provided.
3.	Clear writing and illustrations. The authors have stated their motivations and implement details clearly with well-drawn figures. I find no overstatements in this paper.

Weaknesses:
1.	More experimental datasets would be welcome. The authors have performed molecular property prediction experiments on QM datasets and FreeSolv. More experiments would be welcome on more datasets, such as in MoleculeNet and OGB, to further evaluate the performances of ST-KD on different downstream molecular tasks.
2.	More experiments on parameter analysis would be good. The authors run the knowledge distillation experiments on PCQM4M with a fixed network setting. Additional experiments can be done to explore the effects of different network structure and training hyperparameters.

summary_of_the_review: This paper is based on an interesting idea of building a fast and high-quality SMILES-based molecular fingerprint generator with knowledge distillation from state-of-the-art graph-based models. The authors have described the proposed methods clearly, with their claims being supported by experimental results. Although more experiments can be performed to give a thorough evaluation to the model’s performance, I believe the paper should be recommended due to its novel idea and convincing results.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper targets speeding up the inference process (specifically, SMILES-to-graph preprocessing) in virtual screening by adopting SMILES-based transformer for molecule representation learning. To ensure the performance of SMILES-based transformer, authors propose to use graph-based transformer as teacher network to conduct knowledge distillation (ST-KD). Numerical results claim for the competitive performance and faster inference time.
main_review: I am uncertain about the efficiency claim in this paper. According to my experience, SMILES-to-graph preprocessing is not that time-consuming (in my case probably 10 mins for 10M conversion in 1 process), and this is a preprocessing step, meaning that it only executes once for the future repeated use. Furthermore, a lot of molecule libraries are stored in .sdf format that we only need to read without conversion. I thus feel the motivation is exaggerated while I am also glad to listen to other reviewers.

Besides, it sacrifices the training efficiency to trade the inference one. According to Table 1, compared with GNNs, ST-KD is 3-10 times larger (so what is the training cost which is not reported?) considering training is usually much more expensive than inference. Since graph transformer is also used in training, SMILES-to-graph conversion exists thus the comparison would be training complexity.

I am also concerned with the tokenization step in section 3.2 that authors convert SMILES into a sequence composed of atom characters and bond characters. For instance, cc=cc will be converted to cccc= if I understand correctly. If it is the case, there will be two types of mistakes: 1) different molecules will produce the same token sequences as both cc=cc and c=ccc will be cccc=; 2) the same molecules will produce different token sequences as c=occ and ccc=o will be cocc= and ccco=, respectively. I do not judge on the second mistake since this is essentially the drawback of SMILES representations which is not permutation invariant, but I am worried about the first one.
summary_of_the_review: I am concerned about the over-claim of efficiency and the correctness of tokenization.
review_title: null

Meta-review: The paper  builds  fast and high-quality SMILES-based molecular embeddings  by distilling  state-of-the-art graph-based models teachers.
This has the advantage of speeding inference time w.rt to graph based methods. 

The reviews were split regarding the motivation of the work, in the sense of why not train directly on SMILES instead of distilling graph based methods that are in some tasks behind SMILES transformer. Authors provided clarifications in the rebuttal showing that on Knowledge distillation of graph models  surpasses  SMILES only model training. 

I think given the experimental nature of the paper the main motivation of the paper should be better clarified and supported with more experimentation and downstream tasks.
Decision: Reject