Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: 
This paper provides an end-to-end model, denoted as MS2 -Transformer, using MS/MS spectrum data to specify molecule identity. The application is very important to biochemical studies.  This model consists of three modules, the peak embedding module, the fragmentation aggregation module, and the SMILES reconstruction module. The MS2 - Transformer incorporates chemical knowledge through a fragmentation tree from the MS/MS spectrum. 
 

main_review: The paper is tackle a very interesting biochemistry problem --  tandem mass spectrometry (MS/MS).  The paper provides clear description for the audience to understand the data and the problem (e.g. Fig 4 and Fig5). The three modules are interesting (e.g. encode the peak embedding and incorporate fragmentation tree structure). 

Some questions: 
(1) Although the three modules framework is interestingly designed, it seems to me that the contribution of machine learning novelty seems not very clear. 
(2) Table 1 shows the MS/MS number of pairs seem limited, especially CASMI. Are transformer-based or deep learning models that require a lot of training data the most appropriate methods, compared with non-deep learning based methods?  
(3) MS/MS is a well established technique for many years with mature data analysis pipeline (from spectrum to molecule identification), can the author provide evaluation against currently working data analysis pipeline to Table 2 and Table 3 so that the readers can understand the performance of deep learning model against conventional working model?
(4) Fig 5 a,  why the true predictions in the circles have different scores? 
(5) Fig 3 SMILES-reconstruction module, why the input is smiles (shifted right). Is the smiles string the target, not the input? 

summary_of_the_review: This paper proposes an interesting method to solve an important biochemical problem, data analysis of tandem mass spectrometry (MS/MS). The paper may has a larger scientific impact if it submit to a corresponding high impact journals, which are likely have more appropriate audience. However, if the paper is interested in machine learning conference, some more contribution of machine learning novelty and more quantification of the methods can strengthen the paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposed to a use a deep learning model with a transformer architectur to translate between mass spectra and chemical structures of unknown metabolites.
main_review: This paper addresses an important problem using a convincingly good model.  The design choices overall seem sensible, and the results appear to be strong.
However, I have two main critiques, and a number of minor comments as well.

First, this is a well-studied problem, but the the paper fails completely to situate the performance relative to the state of the art.  For example, we are told (p. 6) which performance measures were used to evaluate the system, but not whether these are the same performance measures employed in other studies (e.g., CAMSI).  It's important to use commonly accepted performance measures or to justify using other measures.  More importantly, the method is compared to the author's own implementations of several competing machine learning models. This is mildly interesting, but not nearly as interesting as would be a comparison to existing, state-of-the-art methods on these same benchmark datasets.  Without such comparisons, it's impossible to evaluate the practical impact of this work.

Second, the description of the peak embedding module (Section 3.1) is not clear at all.  I get that E_{m/z} is a matrix of dimension V \times d, but then I don't understand what E_i is. I think maybe it's a vector of length d, but the sentence is written as if there is a matrix of size E_i \times d. The next sentence says that V is the number of distinct m/z ratios.  I think this means that V is the number of peaks, but then it's also the number of intensities.  I don't understand why V only applies to the m/z ratios (since every m/z ratio comes paired with an intensity). The real problem is that we are told that m_i is encoded using a one-hot representation.  This is completely mysterious to me, and is related to the incorrect claim (p. 3) that m/z is discrete. The m/z ratio is a real value, measured with varying amounts of precision depending on the instrument type.  I do not understand in what sense this measurement can be considered discrete, nor how to encode it as a one-hot representation.  I am guessing that it's a vector where each entry corresponds to a mass bin, but then why aren't we told what bin size is used?  This seems like a critical parameter, and for realistic values it means that the vector will be really long (e.g., hundreds of thousands).  Given all the problems in the first 1.5 sentences of this subsection, I really can't make sense of Equation (1) nor the subsequent description of the encoder.

The claim (p. 1) that "about 90% [of] spectrum molecules is not covered by the standard database" is problematic.  No citation is given to support this claim. Plus, it surely cannot be true in general, since the coverage will depend strgonly on the type of sample being analyzed and the choice of database.

The fragmentation tree shown in Figure 1B is not described well enough.  More detail needs to be provided somewhere in the paper about how the initial graph is constructed and exactly what the edges mean,

At first I thought that Table 1 and Figure 4 seemed redundant and that the table should be eliminated. But then I noticed that the mean values marked in Figure 4 do not agree with the mean values reported in Table 1. So I think I must be misunderstanding something here, which is worrisome.

We are told on p. 8 that the model achieves its performance "in a light-cost [sic] way" using only "an additional 0.26 M parameters." What is this relative to?

The fragmentation inference task in Section 4.4 is not well described. I do not really understand what exactly the problem is here. More importantly, no baseline method is included for this task, so we do not know whether the proposed model improves on the state of the art.

The use of English is problematic throughout.  Careful editing is needed to fix grammatical errors.  Here are a few:

Abstract: "demonstrating it the great"

p. 1: "progresses have been achieved"

p. 5: "uniformed sampled"

In Figure 1A, the fragmentation is not always accomplished via a laser.


summary_of_the_review: I think this might be very good work, but the explanation of the model is problematic, and the empirical results fail to explain how the model's performance compares to the state of the art.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a transformer model for molecule structure identification using the experimental mass spectra. 
This model takes the experimental mass spectrum and the corresponding fragmentation tree structure as the input for the transformer to predict the SMILES of the molecule in an end-to-end manner. 
In their experiments, they show their model improves a vanilla transformer by 5% top-1 accuracy.
main_review: This is an interesting paper that exploits sequence to sequence machine learning models to solve mass spectrometry problems. 
The strength of this paper is that they incorporated extra chemical knowledge of the spectrum via its fragmentation tree to enhance the input feature. 
The weakness of this paper mainly comes from its experimental settings: 
(1) The paper doesn't explain how many different molecules/compounds are in the MassBank dataset and the CASMI dataset. 
(2) How are your training, validation, and testing sets split? The MassBank dataset contains spectrum records of the same molecule but at different experimental conditions (e.g. different collision energy). On average there are around 10 records per compound. Did you split your training, validation, and testing sets randomly or by compounds? This makes a huge difference in terms of performance. The former would be an easy task due to the high overlapping of known compounds in the training set, which makes it more like an overfitting problem. The latter should be the right way to test the model's performance.
(3) How is your reconstruction loss defined? 
(4) What is your label in equation (7)? 
(5) The same molecular structure could have many different SMILES representations. How does your model address this issue (non-deterministic mapping)?
(6) They author doesn't provide a code to reproduce their experiments.
summary_of_the_review: This paper proposed a customized transformer to solve the molecular structure identification problem via MS/MS spectrum. The application itself is interesting but the authors need to justify their performance with a clear experimental setting and a better illustration of the evaluation process. 
With the current paper, I this it is marginally below the acceptance threshold. If you address my questions properly, I may adjust my score.
review_title: null

Meta-review: Finally, all reviewers leaned towards rejection. The main concerns were missing methodological depth and questions regarding the experimental evaluation (unclear link between experimental outcomes and methodological details). The rebuttal was not perceived as being fully convincing, and finally nobody wanted to champion this paper. I think that this work has some potential, but in its present form, it does not seem to be ready for publication.
Decision: Reject